Deer Detection Diversion 1 – Detecting Day and Night

In a previous post, I wrote about how I wanted to use Machine Learning to detect when a deer is present in my back yard, so that I can make sure that no deer-built weapons of mass destruction are constructed in my yard (at least not on my watch!). After initially using a Microsoft 360 Kinect for my first data collection phase, I ran into a problem: deer (and many other animals) are actually nocturnal. The Kinect camera, while easy to use, doesn’t have a sensitive enough IR camera to see in the darkness of my backyard, even with the help of some nifty IR emitters.

To solve the problem of night vision, I wrote about how I set up a Raspberry Pi with a NOIR camera to keep track of the murderous deer in the yard both day and night. Here’s a picture of the new rig:

rpi_ir_camera_rig

My NOIR camera and Raspberry Pi. Plus my super low cost NOIR case – an SD  card holder.

The Challenge

While the camera and IR emitters work great for capturing pictures, I ran into a timing problem. The NOIR camera requires very different settings between day-time and night-time exposures.

During the day time, I can use the camera’s automatic white balance feature, along with an ISO of 100. However, at night I need to tweak the camera to use a very specific white balance, and adjust the shutter speed so that it stays open for nearly 3 seconds.

The main challenge comes when it is time to swap from day-mode to night-mode automatically. If I leave the NOIR camera on night mode for too long into the sunrise, I get images like this:

noir_night_dawn

Whoa! Is it a nuclear explosion with a dreaded EMP!? Oh… no, it’s just dawn.

Right now, I set the camera on a schedule. Using a small Python script, I get the camera to take a set number of pictures before stopping, and then swap settings from day to night, or night to day. This works okay, but there is one glaring problem – the sun rises and sets at a different time every day! Pesky sun!

The Solution

While I could solve the problem by hooking into a web service that tells me when the sun is about to rise or set, I think a more interesting solution is to have the Python script look at each exposure, and swap the day and night settings by itself.

Note that this sounds like I’m proposing some sort of self aware program… but I’m not talking about Skynet levels of intelligence – I don’t need to be avoiding murderous deer and a runaway, self-aware, weakly god-like super-intelligent Raspberry Pi. I’m thinking more along the lines of using color histograms. Nice, simple, and (usually) non-deadly color histograms.

Too Bright or Too Dark?

The problem of when to transition between camera modes boils down to sampling an image and calculating the average pixel intensity. When it’s dark out, most of the image will become very dim, resulting in a color histogram that is close to black. When it becomes light outside, the opposite is true – the color histogram will be close to white.

So, how does one actually compute a color histogram? It’s easy! Usually, images are stored (or can be transformed into) an RGB format where each pixel has 3 color components: R (red), G (green), and B (blue). Each of these components has an intensity value that will usually take on a value between 0 and 255. For example, if there was a pure red pixel, its color value might be R = 255, G = 0 and B = 0. Pure black is R = 0, G = 0, and B = 0. Pure white is R = 255, G = 255, and B = 255. A nice brown deer color would be R = 130, G = 88, and B = 64.

To compute a color histogram, the program needs to loop through each pixel in an image and read the red (R), green (G) and blue (B) intensities, summing up the totals for each intensity level. The result is something like this:

Intensity |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
 0        | 0        | 0          | 0          |
 1        | 0        | 0          | 0          |
 2        | 0        | 0          | 0          |
 3        | 0        | 0          | 0          |
 ...      | ...      | ...        | ...        |
 237      | 8101     | 5949       | 5940       |
 238      | 15527    | 8518       | 8842       |
 239      | 214433   | 13467      | 16682      |
 240      | 24592    | 24301      | 208580     |
 241      | 38842    | 50694      | 59539      |
 242      | 7938     | 183090     | 14318      |
 243      | 6045     | 750        | 5157       |
 244      | 2121     | 8          | 3872       |
 245      | 1550     | 0          | 2063       |
 246      | 779      | 0          | 1280       |
 247      | 368      | 0          | 1022       |
 248      | 311      | 0          | 721        |
 249      | 168      | 0          | 451        |
 250      | 132      | 0          | 350        |
 251      | 84       | 0          | 257        |
 252      | 80       | 0          | 207        |
 253      | 40       | 0          | 149        |
 254      | 28       | 0          | 125        |
 255      | 73       | 0          | 469        |

This is actual histogram from my nuclear dawn picture I showed above. I’ve truncated the data above and cut out intensities 4 – 236. Interpreting the chart is fairly straightforward. For example, there are 183,090 pixels that have a G value of 242, meaning that there is a lot of high green values. Clustered around the 240 intensity level are the bulk of the RGB values, which I would expect if the picture has a lot of white in it (which it does).

Actually accessing the RGB data in a program is easy – it turns out the OpenCV project has a Python library called cv2 that makes this possible:

import cv2
image = cv2.imread(filename)
blue = cv2.calcHist([image], [0], None, [256], [0, 256])
green = cv2.calcHist([image], [1], None, [256], [0, 256])
red = cv2.calcHist([image], [2], None, [256], [0, 256])

This produces the histogram information for the R, G, and B components of the image. What is more useful is to plot the histograms out to give you a feel for where the main bulk of the color intensities are located. MatPlotLib is the Python package to use. Here is what the histogram data look like:

color_histogram_nuclear_dawn

The color histogram for the nuclear dawn picture. Notice how the red, green and blue components are all really high and clustered together! This is important!

Now, let’s compare it to another picture. This is one where the camera was running in day mode, and it started to get dark out. Here is the picture:

day_night_transition

The world is going dark! Oh noes! What happened to the sun!?

Now let’s take a look at the histogram data:

color_histogram_darkness

This is the histogram for the picture where the sun is burning out! Ahem… I mean where the sun is setting… we probably have a few billion years before it actually burns out.

Notice that the histogram data on both images tends to polarize to one end of the spectrum or another – for the nearly white picture, the bulk of the components are near 255. For the dark image, the bulk are near zero.

I can use this to my advantage in the Python program. All I have to do is see if the intensity values for each color component are near 0 or 255. However, for this to work, I need to calculate the weighted mean of each of the R, G, and B components, and compare them to some cutoff values.

The weighted mean calculation works similar to the usual mean calculation. The difference is that instead of just summing up all of the counts for say the R component, I multiply the count by the intensity value, perform the sum, and then divide by the total number of pixels we have. So, for the R component, the calculation becomes:

R_weighted_sum = ((0 * R[0]) + (1 * R[1]) + ... (255 * R[255])) / sum(R)

I just repeat that weighted sum calculation for the G, and B components. For example, the weighted values for the nuclear dawn photo is as follows:

          |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
Wgt-Mean  | 238      | 237        | 240        |

Putting It All Together

All that I needed now was to get my Python camera implementation to check for histogram data every few snapshots, and calculate the weighted sum of intensities. To get it to transition into day mode, I look for the average intensities of the R, G and B values to be greater than 230. For the night mode, I look for the average intensities to be less than 30. When it detects either of those situations, it will adjust the camera settings accordingly.

As an example, I created a time-lapse view from 6:00 am to 6:12 am. The program bounces back and forth a little between modes until it becomes bright enough in day mode to use the automatic settings (note that the animation loops!). I can tweak this a little by using slightly different cutoffs for the pixel intensities:

transition

Time-lapse view of the transition from night mode to day mode. The flicker back and forth is because it’s still too dark for day mode. As the sun gets brighter, the program stabilizes in day mode.

The full source code for the program is available in my GitHub repository. Check it out and feel free to experiment with it!

Summary

Using histogram data to swap camera modes is very easy. I showed how to compute the histogram data, and calculate the weighted mean of the intensity values. With both of these tools in hand, it’s easy to determine whether the camera should switch to night mode or day mode based on the light values! As long as the deer doesn’t figure out how to blow up the sun, I can leave my Raspberry Pi camera program running until I run out of disk space. This will collect good evidence if the deer manages to murder me, or, more likely, eats all of my vegetables.

Check back soon when I will (finally) discuss the Machine Learning component of the project.

Deer Detection with Machine Learning (Part 2)

ir_test_night

Last time I wrote about building a simple Java program to take pictures of my backyard to keep track of those murderous deer that infest our backyard. To recap, I want to use machine learning to teach a computer to recognize that this is a deer:

This is a deer.

This is a deer.

And teach the computer that this is not a deer:

This is not a deer.

This is not a deer.

When I last wrote, I was going to have the computer take pictures every 10 seconds via a Kinect, and then create my training data set by finding lots of pictures with deer in them, and lots of pictures that were deer free. However, what I thought would be a walk through the park ended up being a walk through a scary, deer-absent park of horrors instead.

A Watched Yard Has No Deer…

I have no idea if the deer has a good data plan and read my blog (antlers make good cell receivers), or if it has some form of ESP and could read my thoughts – in either case, after 7 days of observation I have too few examples of deer for my machine learning approach to work. This is despite the fact that there is mounting evidence (i.e. deer poop – the jerk) that the deer is in the yard quite often.

A second or two on the internet provided me a fairly good answer. As it turns out, deer are primarily nocturnal. This means I’m watching at the wrong time of day. I find this very strange, since we’ve seen the deer in our yard quite often during the daytime. Luckily, the Kinect has a second camera – one that sees in infrared.

Note: actually, it’s not entirely true that I saw NO deer. I actually did catch a few glimpses of them. The problem was that there were too few images to use for training. Here’s an example of one of them:

2014-08-18_05-58-44-deer-legs

A picture of the deer – if you squint, you can see the deer’s legs in the picture. Obviously, this deer is very camera shy.

Alone in the Dark

To take depth images, the Kinect uses an infrared camera, and a cool infrared laser (which I will call the IR emitter from here on in) that shoots out a pattern of dots. The IR camera uses the dot pattern from the IR emitter to figure out how close or far away objects in the image are.

Using the OpenKinect libfreenect wrappers, it is actually quite easy to take an IR picture. When taking a video, specify the video format as VideoFormat.IR_8BIT. Each frame will have a ByteBuffer where a single byte represents the light intensity of a single pixel. To convert that into grayscale, just set the R, G and B components to the byte value. The following Java function will convert the ByteBuffer properly:

private BufferedImage getIR8BITBufferedImage() {
    int width = sFrameMode.width;
    int height = sFrameMode.height;
 
    // Convert the ByteBuffer into a ByteArrayInputStream for easier access
    byte[] data = new byte[sByteBuffer.remaining()];
    sByteBuffer.get(data);
    ByteArrayInputStream stream = new ByteArrayInputStream(data);
 
    mBufferedImage = new BufferedImage(width, height, 
                BufferedImage.TYPE_BYTE_GRAY);
 
    // Loop through the image data and write it into the BufferedImage
    for (int y = 0; y < height; y++) {
        for (int x = 0; x < width; x++) {
            int g = stream.read();
            mBufferedImage.setRGB(x, y, new Color(g, g, g).getRGB());
        }
    }
    return mBufferedImage;                
}

To test it out, I put the camera in a window and snapped a few IR photos during the daytime. After some tweaking, I got it working really well. Originally I had problems – it turns out that the dot pattern generated by the IR emitter reflects in the window, causing a glare. After covering the IR emitter with a tea towel, and framing the Kinect with a similar material to cut down on room glare, the resulting images looked like this:

An IR photo snapped during the daytime – ooooh, aaaah.

So, how well does it work at night? Here’s the result:

ir_test_2

Here’s an image that the Kinect took at night through the window. Notice the subtle variations of black throughout the frame.

The picture is the same when taken through the window, and through the screen. When I uncovered the IR emitter, the result was the glare of the IR dot pattern against the window. Unfortunately, without a source of IR light in the 830nm wavelength illuminating the yard, the IR camera isn’t able to pick up anything. I blame the deer.

Stepping into the Limelight

The solution was to find an infrared illumination source. I managed to find some cheap CCTV IR emitters on eBay. While they are of the 850nm variety, there is enough bleed across other wavelengths for the Kinect IR camera to pick it up (in fact, they glow faintly red, meaning that they bleed all the way into the 700nm range!). Here’s a picture of one of the IR emitters:

ir_emitter

One of the IR emitters I bought on eBay. It also has a nifty photo-sensitive cell in it, so it only turns on at night.

Note: many people have problems with these, because the box doesn’t list the power supply requirements. For the record, it is 12V DC, 500 mA, barrel center positive. With a little digging, you can pick a used one up at your local Value Village.

So, after waiting a week for my IR emitters to arrive, I plugged them in, aimed them out the window, and turned on the camera at night to see what happened.

And…

Still nothing. Just a black image. To double check that the Kinect could actually see the light being cast by the IR emitters, I turned them on in my room, and pointed the camera at me. My image came back lit very brightly, even though I was sitting in the dark.

Not to give up so easily, I took the IR emitters and the Kinect outside, pointed everything straight ahead, and had my DW stand roughly 10 feet in front of the rig. Here’s the result:

ir_test_led_rebalance

My DW about 10 feet away from the Kinect and IR emitters. I’ve actually enhanced the image so you could see what the Kinect picked up.

If you look closely, you can just about see her outline in the image. Note that she isn’t a deer. But there could be one lurking behind her – we’ll never know!

The verdict: the IR camera on the Kinect isn’t sensitive enough to the light being cast by the IR emitters. I was hoping that there would be a lot of bleed-through to the 830nm wavelength the Kinect used, but alas, it just wasn’t happening. That isn’t good, since every night that passes means another night where the deer might attempt to murder me.

So, we’re going to need a bigger boat… er… smaller camera.

Easy as Raspberry Pi

A while ago I bought a little device called a Raspberry Pi. For those of you who don’t know, it’s a neat little computer on a single circuit board that is about the size of a business card.

pi_next_to_card

One of my early model B Raspberry Pi devices in a case, next to a business card. It’s slightly bigger than a business card. Well, bigger and warmer than a business card – and pokier. Plus it needs power.

The Raspberry Pi is a very cool low power computer that runs various operating systems, and has many different input and output options available for it. One new development board recently released for the Pi is the NOIR camera. The NOIR actually stands for “no IR” – meaning the IR filter has been removed from the camera. Most normal cameras have an IR filter on them so that you only see the wavelengths that you are used to seeing through your own wetware (i.e. your eyeballs). Without an IR filter, the pictures you get back would actually look a little strange.

A Film NOIR

The NOIR camera plugs into the Raspberry Pi board, and with a few tweaks to the OS, is easy to get up and running as a camera. To enable the NOIR camera, simply run:

sudo raspi-config

Of the options on the screen, one of them will let you enable / disable the Raspberry Pi camera module. With that done, all I needed to do was mount the camera, fiddle with the exposure time and white balance settings through Python using the picamera module.

Note: I could have used the raspistill command line program, but it has the annoying habit of turning on the red LED for the camera when it takes a picture. Plus I want to control some other aspects of the camera. I created a GitHub repository with the script I used to turn off the LED while taking pictures.

Here is a picture of the rig with the IR camera:

rpi_ir_camera_rig

The finished rig pointed out the back window.

Inside the white case is a newer Raspberry Pi model B. I used an SD card case as a case for the NOIR camera. The washcloth underneath is to stop the glare from the window. I actually put another one on top of it as well (image olde-timey cameras – the person taking the picture needed to stand under a black cover – same thing here to cut down on glare). The Raspberry Pi also has a wireless USB stick in it, so that it can save the pictures to a NAS device I have on our network.

Night Test

After running through several tests, I waited for nightfall and then I carefully aimed both IR emitters out the window, and snapped a photo (well, many actually – it involved a lot of readjustment for both the camera and the IR emitters). Here’s one of my sample pictures:

ir_test_night

An IR photo of me taken at nearly 1 in the morning. Yes, I’m wearing a hat – I’m a redhead. I also felt very silly standing in pitch blackness waving at the camera.

There is enough exposure to clearly see everything out in the back yard! Excellent! No deer can hide now!

The only caveat to this is that I’m badly over-exposing each frame I take in order to get enough IR light to the camera. To give you a sense of the over-exposure – the shutter on the camera remains open for nearly 6 seconds. The result is that any moderate or fast motion will be very blurry. I’ll try a few more experiments with it to see if I can get a better image.

Conclusions

The deer is still out there… somewhere… plotting my murder. Unfortunately, the Kinect IR camera isn’t sensitive enough to the light thrown by my IR emitters. Luckily the Raspberry Pi NOIR camera sees very well in the dark. For now, it’s back to data collection mode both during the day, and at night. Mu-ha-ha-ha!

Check back soon, where I’ll actually start describing the Machine Learning component I’m hoping to use. I’ll try it out on another shady backyard visitor – one who wears a mask and steals our blackberries!