Deer Detection with Machine Learning Part 2

Last time I wrote about building a simple Java program to take pictures of my backyard to keep track of those murderous deer that infest our backyard. To recap, I want to use machine learning to teach a computer to recognize that this is a deer:

And teach the computer that this is not a deer:

When I last wrote, I was going to have the computer take pictures every 10 seconds via a Kinect, and then create my training data set by finding lots of pictures with deer in them, and lots of pictures that were deer free. However, what I thought would be a walk through the park ended up being a walk through a scary, deer-absent park of horrors instead.

A Watched Yard Has No Deer

I have no idea if the deer has a good data plan and read my blog (antlers make good cell receivers), or if it has some form of ESP and could read my thoughts - in either case, after 7 days of observation I have too few examples of deer for my machine learning approach to work. This is despite the fact that there is mounting evidence (i.e. deer poop - the jerk) that the deer is in the yard quite often.

A second or two on the internet provided me a fairly good answer. As it turns out, deer are primarily nocturnal. This means I'm watching at the wrong time of day. I find this very strange, since we’ve seen the deer in our yard quite often during the daytime. Luckily, the Kinect has a second camera - one that sees in infrared.

Note: actually, it’s not entirely true that I saw NO deer. I actually did catch a few glimpses of them. The problem was that there were too few images to use for training. Here’s an example of one of them - if you squint, you can see the deer's legs in the picture. Obviously, this deer is very camera shy.

Alone in the Dark

To take depth images, the Kinect uses an infrared camera, and a cool infrared laser (which I will call the IR emitter from here on in) that shoots out a pattern of dots. The IR camera uses the dot pattern from the IR emitter to figure out how close or far away objects in the image are.

Using the OpenKinect libfreenect wrappers, it is actually quite easy to take an IR picture. When taking a video, specify the video format as VideoFormat.IR_8BIT. Each frame will have a ByteBuffer where a single byte represents the light intensity of a single pixel. To convert that into grayscale, just set the R, G and B components to the byte value. The following Java function will convert the ByteBuffer properly:

private BufferedImage getIR8BITBufferedImage() {
    int width = sFrameMode.width;
    int height = sFrameMode.height;

    // Convert the ByteBuffer into a ByteArrayInputStream for easier access
    byte[] data = new byte[sByteBuffer.remaining()];
    sByteBuffer.get(data);
    ByteArrayInputStream stream = new ByteArrayInputStream(data);

    mBufferedImage = new BufferedImage(width, height, 
                BufferedImage.TYPE_BYTE_GRAY);

    // Loop through the image data and write it into the BufferedImage
    for (int y = 0; y < height; y++) {
        for (int x = 0; x < width; x++) {
            int g = stream.read();
            mBufferedImage.setRGB(x, y, new Color(g, g, g).getRGB());
        }
    }
    return mBufferedImage;                
}

To test it out, I put the camera in a window and snapped a few IR photos during the daytime. After some tweaking, I got it working really well. Originally I had problems – it turns out that the dot pattern generated by the IR emitter reflects in the window, causing a glare. After covering the IR emitter with a tea towel, and framing the Kinect with a similar material to cut down on room glare, the resulting images looked like this:

So, how well does it work at night? Here’s the result:

The picture is the same when taken through the window, and through the screen. When I uncovered the IR emitter, the result was the glare of the IR dot pattern against the window. Unfortunately, without a source of IR light in the 830nm wavelength illuminating the yard, the IR camera isn't able to pick up anything. I blame the deer.

Stepping into the Limelight

The solution was to find an infrared illumination source. I managed to find some cheap CCTV IR emitters on eBay. While they are of the 850nm variety, there is enough bleed across other wavelengths for the Kinect IR camera to pick it up (in fact, they glow faintly red, meaning that they bleed all the way into the 700nm range!). Here’s a picture of one of the IR emitters:

Note: many people have problems with these, because the box doesn't list the power supply requirements. For the record, it is 12V DC, 500 mA, barrel center positive. With a little digging, you can pick a used one up at your local Value Village.

So, after waiting a week for my IR emitters to arrive, I plugged them in, aimed them out the window, and turned on the camera at night to see what happened.

And...

Still nothing. Just a black image. To double check that the Kinect could actually see the light being cast by the IR emitters, I turned them on in my room, and pointed the camera at me. My image came back lit very brightly, even though I was sitting in the dark.

Not to give up so easily, I took the IR emitters and the Kinect outside, pointed everything straight ahead, and had my DW stand roughly 10 feet in front of the rig. Here's the result:

If you look closely, you can just about see her outline in the image. Note that she isn't a deer. But there could be one lurking behind her - we'll never know!

The verdict: the IR camera on the Kinect isn’t sensitive enough to the light being cast by the IR emitters. I was hoping that there would be a lot of bleed-through to the 830nm wavelength the Kinect used, but alas, it just wasn't happening. That isn't good, since every night that passes means another night where the deer might attempt to murder me.

So, we’re going to need a bigger boat... er... smaller camera.

Easy as Raspberry Pi

A while ago I bought a little device called a Raspberry Pi. For those of you who don’t know, it’s a neat little computer on a single circuit board that is about the size of a business card. Here's one of my early model B Raspberry Pi devices in a case, next to a business card. It's slightly bigger than a business card. Well, bigger and warmer than a business card - and pokier. Plus it needs power.

The Raspberry Pi is a very cool low power computer that runs various operating systems, and has many different input and output options available for it. One new development board recently released for the Pi is the NOIR camera. The NOIR actually stands for "no IR" – meaning the IR filter has been removed from the camera. Most normal cameras have an IR filter on them so that you only see the wavelengths that you are used to seeing through your own wetware (i.e. your eyeballs). Without an IR filter, the pictures you get back would actually look a little strange.

A Film NOIR

The NOIR camera plugs into the Raspberry Pi board, and with a few tweaks to the OS, is easy to get up and running as a camera. To enable the NOIR camera, simply run:

sudo raspi-config

Of the options on the screen, one of them will let you enable / disable the Raspberry Pi camera module. With that done, all I needed to do was mount the camera, fiddle with the exposure time and white balance settings through Python using the picamera module.

*Note: I could have used the raspistill command line program, but it has the annoying habit of turning on the red LED for the camera when it takes a picture. Plus I want to control some other aspects of the camera. I created a GitHub repository with the script I used to turn off the LED while taking pictures.

Here is a picture of the rig with the IR camera:

Inside the white case is a newer Raspberry Pi model B. I used an SD card case as a case for the NOIR camera. The washcloth underneath is to stop the glare from the window. I actually put another one on top of it as well (image olde-timey cameras – the person taking the picture needed to stand under a black cover - same thing here to cut down on glare). The Raspberry Pi also has a wireless USB stick in it, so that it can save the pictures to a NAS device I have on our network.

Night Test

After running through several tests, I waited for nightfall and then I carefully aimed both IR emitters out the window, and snapped a photo (well, many actually - it involved a lot of readjustment for both the camera and the IR emitters). Here's one of my sample pictures of me taken at nearly 1 in the morning. Yes, I’m wearing a hat - I'm a redhead. I also felt very silly standing in pitch blackness waving at the camera.

There is enough exposure to clearly see everything out in the back yard! Excellent! No deer can hide now! The only caveat to this is that I'm badly over-exposing each frame I take in order to get enough IR light to the camera. To give you a sense of the over-exposure – the shutter on the camera remains open for nearly 6 seconds. The result is that any moderate or fast motion will be very blurry. I’ll try a few more experiments with it to see if I can get a better image.

Conclusions

The deer is still out there somewhere, plotting my murder. Unfortunately, the Kinect IR camera isn't sensitive enough to the light thrown by my IR emitters. Luckily the Raspberry Pi NOIR camera sees very well in the dark. For now, it's back to data collection mode both during the day, and at night. Mu-ha-ha-ha!

Check back soon, where I'll actually start describing the Machine Learning component I’m hoping to use. I’ll try it out on another shady backyard visitor – one who wears a mask and steals our blackberries!