Deer Detection with Machine Learning (Part 4)

It’s time again for another episode of “Murderous Deer Machine Learning Mayhem”. Last time, I used a neural network to detect raccoons, since they frequented my backyard more often in the early days. In that post, my preliminary results revealed an accuracy (F1 measure) of approximately 72%. In this post, I’ll talk about some of the experiments I performed to try and increase the overall effectiveness of the neural network.

Seeing in Color

My first observation I thought was fairly obvious – color information should lead to better performance. This is an example where my gut feeling and a quick and dirty exploration of the data set seemed to point in a promising direction. Notice the word seemed – that’s because I ultimately went down a bad path. This is a perfect example of why you should dig into the data much more – and have a better grounding in the domain – before investing too much time into feature engineering. Past-self was warning current-self about this in previous blog posts here and here. Bad Craig for not listening, bad! But before I reveal why this didn’t work, let me describe how I adapted the architecture of the neural net to see in color.

Changing the Network

Background – one type of computer image ultimately boils down to a combination of three different color components – Red, Green and Blue (RGB). By mixing these components together with varying intensities of each, you can produce a wide variety of colors. For example, if each component can take on an intensity value between 0 and 255, you can produce 16,777,216 different colors. Cool!

My idea for the neural  network was simple – break down each pixel into its component values, and feed all of that to the neural network. A picture will probably describe this much better. My original network looked like this, where I took the average of the RGB intensities and stored it in a single value (creating a grayscale image):

network_diagram

I modified the network (by modify, I mean built in an additional option to the program) to subdivide each pixel into its R, G and B component, and fed that to the neural network like so:

network_diagram_color

This means that I now had 3,600 x 3 = 10,800 inputs to the neural network. A bit of a combinatorial explosion in the number of network connections, but heck, memory and computing power is cheap these days.

Some Experiments

I ran quite a number of experiments on the raccoon data with more control on how testing and training data sets were built, both to re-affirm my original results (which hold, ~70% accuracy is right on the button), and to try out a few different options, which included color information. I’ll explain the different options in a moment, but first, here are some results.  Note, these results are after training the network over 1,500 iterations to convergence, and then performing a 10-fold cross validation.

Precision

Precision (also known as the positive predictive value) is a measurement of how well the machine learning algorithm is doing when it labels something as a positive example. You basically look at all of the things that the classifier labels as a raccoon, and see how many of them are actually raccoons. High precision is good – it means the algorithm is accurately identifying positive examples.

color-raccoons-precision

In the case of the Baseline, you can see that it does fairly well – roughly 70% of the things it labelled a raccoon were actually raccoons. This is good. Notice however, that the Hyperbolic did better (more on that in a bit). Notice too that the precision with Color information was significantly lower.

Recall

Recall (also known as sensitivity) measures how well the algorithm is remembering the actual examples it was given (I always think of the movie Total Recall – the original 90’s version with the Governator). Put another way, recall asks: of all of things that were actually raccoons, how many did it properly identify? As an example, if there are 10 raccoons in the data set, and the algorithm correctly identifies 6 raccoons, then the recall is 60% (total recall would be identifying all 10 raccoons).

color-raccoons-recall

As we can see, the Baseline recall is roughly 70%, with the Hyperbolic performing equally. Again, it took a hit with Color, but this time, the Hyperbolic Color does better. Notice the variance between the different options (the black lines that indicate a range). Both Color and Hyperbolic Color have quite significant variances. This means that those models are fairly fragile.

F1 Measure

The F1 measure is a way of combining both the Precision and Recall of a model into a single measurement (also known as a harmonic mean). Why? Well, sometimes it’s nice to be able to judge performance based on a single feature, rather than comparing the features separately. In a previous post, I used the F1 measure and accuracy to mean the same thing.

color-raccoons-f1

As you can see here, the Hyperbolic wins the race for overall performance. Let’s talk about what is happening here.

The Hyperbolic Tangent

In a previous post, I talked about how neural networks work. One of the key features is the activation function for the network. When a signal is passed from one end of the network to the other, it must pass through several layers in the network. The signal must be a certain “strength” before it is passed on. The thing that measures the strength and “decides” whether to pass on a new signal is the activation function.

The Hyperbolic Tangent is an activation function with a slightly different profile than the one I was using in the Baseline model. The Baseline model uses the Sigmoid as an activation function. Here is a plot of how the two functions stack up with some values between -10 and 10:

sigmoid_tanh

As you can see, the Hyperbolic Tangent maps values between -1 and 1, and is triggered slightly differently than the Sigmoid.

Long story short, recall for both the Hyperbolic Tangent and Sigmoid models were basically identical. The difference was with precision. Basically, the Hyperbolic Tangent resulted in a model that was more precise than the Sigmoid model, which is why the F1 measure is slightly better (because it is a combination of both precision and recall). Note that both the Baseline and Hyperbolic models were produced using grayscale images. So, what is going on with color?

Color Made Things Worse

Yup. That’s right. Performance tanked with color information (okay, it’s not abysmal, but it is considerably worse than my original run with grayscale images). At first I thought I had an error somewhere in my code. But looking at the data more, as well as false positives and false negatives, I started to hypothesize that the way I was using color information was probably hurting me very badly. The reason – I think – is due to the fact that I’m losing shape information when I separate out each color band.

To explain a bit more, in both the Baseline and Hyperbolic (without Color) models, for each pixel in each image, I take an average of all the different color bands. This does two things:

  1. It converts the color image into a grayscale image.
  2. It normalizes and accentuates edges.

Point 2 I think is the important part. To demonstrate this a bit more clearly, I looked at the following false negative (what was a raccoon, but what the algorithm thought was not):

fn37

I separated it into its R, G, and B components and looked at the intensity of each component.

Here is the R component:

red_fn37

 

Here is the G component:

green_fn37

 

Here is the B component:

blue_fn37

Notice how difficult it is to make out shapes when you treat each color band separately (especially the blue component). The raccoon tends to blend in with the background. Aside from the white on its face and front, it is very difficult to make it out. For a comparison, here is what the picture looks like if I convert it to a proper grayscale:

i-fn37

Notice how it becomes much easier to separate the actual shape of the raccoon out of the background.

The long and the short of it is while I thought I was giving the neural network more data to work with, in actual fact, I was making it harder for the network to distinguish shapes. I need to do some reading to find a new strategy for dealing with color information. Feel free to reply in the comments section if you have an idea for an approach!

In the Meantime – Deer!

With some preliminary analysis out of the way, and with a functioning neural network, I think it is finally time to perform deer detection. I now have over 5,500 images of deer to train with. It will take some time to process that many images in order to generate a training set. Half of those images are taken during the night, and half during the day. Many of them are of the deer laying down, like so:

deer_laying_downNotice that pesky post in the way of the deer! Plus, eventually those deer are going to lose their antlers (ha-ha – one less weapon with which to murder me). All in all, there are going to be some unique challenges to identifying deer.

Wrapping Up

In this post, I talked about the performance of the neural network with respect to identifying raccoons, and talked about why color information didn’t work out as I originally planned. Tune in next time when I’ll look at identifying deer and some of the challenges that brings.

Gradle and SonarQube

If you are like me and like developing using a Test Driven Development (TDD) approach, then you need the ability to examine your code and test coverage. In a past article, I discussed how to use SonarQube to perform Python code inspection allowing you to see code test coverage. In this article, I’ll look at how to set up Gradle to talk to SonarQube.

The SonarRunner

SonarQube has a companion tool called the SonarRunner. The runner is what actually runs the unit tests (or integration tests) on your software, and then reports it to the SonarQube database. One of the nice things about the SonarRunner is that you can add it to just about any project that can be analyzed by SonarQube – regardless of whatever other lifecycle or build tools you may use. All you need to do is add a sonar-runner.properties file with the required information.

One problem however, is that the SonarRunner is yet another tool that is needed during the build process. If you maintain a continuous integration environment where, say, your nightly builds include an inspection run, then you know the pain of having to maintain yet another standalone binary package. Luckily, Gradle has a SonarRunner plugin that is incubating, and is stable enough to use.

Setting Up the Gradle Plugin

The SonarRunner plugin has been included in most recent distributions of Gradle. Applying the plugin requires one line:

apply plugin: sonar-runner

In order to get the runner talking to your local installation of Sonar, you will also need a sonarRunner configuration section in your build.gradle file (note – if you have an older version of Gradle this may not work for you, since older versions used slightly different properties). A minimal set of configuration properties is as follows:

sonarRunner {
    sonarProperties {
        property "sonar.host.url", "http://10.0.0.60:9000"
        property "sonar.jdbc.url", "jdbc:postgresql://10.0.0.60:5432/sonar"
        property "sonar.jdbc.driverClassName", "org.postgresql.Driver"
        property "sonar.jdbc.username", "mySonarUsername"
        property "sonar.jdbc.password", "mySonarPassword"
        property "sonar.projectKey", "MyJavaProject"
        property "sonar.projectName", "My Java Project"
        property "sonar.projectVersion", android.defaultConfig.versionName
        property "sonar.language", "java"
        property "sonar.sources", "src/main"
        property "sonar.binaries", "build"
    }
}

Here are some of the settings in more detail:

  • sonar.host.url – this is the URL to your Sonar host. Note that if you are trying to use a host that is protected by SSL, you will need some additional configuration information.
  • sonar.jdbc.url – this is the JDBC (database) URL. In my case, I was using PostgreSQL server, so I used the postgresql JDBC syntax. The 10.0.0.60 is my internal IP address, and 5432 is the port that the database listens on. Note that in the case of PostgreSQL, you need to explicitly tell it the addresses to listen on in your postgres.conf file, otherwise it defaults to listening on port 5432 only for localhost.
  • sonar.jdbc.driverClassName – should be fairly obvious that this is the driver it should be using for the database connection.
  • sonar.jdbc.username – this is the username used to access the database – not a username that you set up to access the front-end application (if you have authentication for SonarQube turned on).
  • sonar.jdbc.password – this is the password used to access the database – again, not a front-end application username password.
  • sonar.projectKey – a unique name used to identify the project.
  • sonar.projectName – the name of the project, more readable for us humans.
  • sonar.projectVersion – should be fairly obvious. In this instance, I have a second set of `android` properties which are describing my project. Instead of duplicating the version number, I just used it here instead.
  • sonar.language – tells Sonar what language you are analyzing.
  • sonar.sources – where the source files for the project are located. Make sure you keep this path separate from your test cases, otherwise you will have your tests included in the analysis.
  • sonar.binaries – where build artifacts are located.

Configuring JaCoCo

While the above configuration will get you up and running, you will be missing coverage information from your unit tests. To generate coverage reports, you will also need to apply the JaCoCo plugin:

apply plugin: jacoco

Then, you need to set two more Sonar Runner properties as follows (add them below sonar.binaries):

property 'sonar.jacoco.reportPath', "${buildDir}/jacoco/testDebug.exec"
property 'sonar.junit.reportsPath', "${buildDir}/test-results"

You may have to adjust these depending on where jacoco is actually putting your results. Look for the exec file and the test-results paths after you run a build.

All Together

Putting it all together:

apply plugin: sonar-runner
apply plugin: jacoco
 
sonarRunner {
    sonarProperties {
        property "sonar.host.url", "http://10.0.0.60:9000"
        property "sonar.jdbc.url", "jdbc:postgresql://10.0.0.60:5432/sonar"
        property "sonar.jdbc.driverClassName", "org.postgresql.Driver"
        property "sonar.jdbc.username", "mySonarUsername"
        property "sonar.jdbc.password", "mySonarPassword"
        property "sonar.projectKey", "MyJavaProject"
        property "sonar.projectName", "My Java Project"
        property "sonar.projectVersion", android.defaultConfig.versionName
        property "sonar.language", "java"
        property "sonar.sources", "src/main"
        property "sonar.binaries", "build"
        property 'sonar.jacoco.reportPath', "${buildDir}/jacoco/testDebug.exec"
        property 'sonar.junit.reportsPath', "${buildDir}/test-results"
    }
}

You can now check out whether or not Gradle is configured correctly by looking at the tasks:

./gradlew tasks

You should see a new task called sonarRunner:

sonarRunner - Analyzes project ':app' and its subprojects with Sonar Runner.

To generate reports, you do the following:

./gradlew clean test sonarRunner

Gradle will clean the project, rebuild and run the unit tests. It will then perform a Sonar analysis, and post the results to your SonarQube instance. If everything is successful, you should see output similar to the following from Gradle:

17:30:40.937 INFO  - Sensor JaCoCoSensor...
17:30:40.965 INFO  - Analysing /home/thomas/AndroidStudioProjects/MyJavaProject/app/build/jacoco/testDebug.exec
17:30:41.598 INFO  - No information about coverage per test.
17:30:41.598 INFO  - Sensor JaCoCoSensor done: 661 ms
17:30:42.955 INFO  - Execute decorators...
17:30:45.010 INFO  - Store results in database
17:30:45.133 INFO  - ANALYSIS SUCCESSFUL, you can browse http://10.0.0.60:9000/dashboard/index/MyJavaProject
17:30:45.205 INFO  - Executing post-job class org.sonar.plugins.core.issue.notification.SendIssueNotificationsPostJob
17:30:45.210 INFO  - Executing post-job class org.sonar.plugins.core.batch.IndexProjectPostJob
17:30:45.266 INFO  - Executing post-job class org.sonar.plugins.dbcleaner.ProjectPurgePostJob
17:30:45.289 INFO  - -> Keep one snapshot per day between 2014-09-19 and 2014-10-16
17:30:45.291 INFO  - -> Keep one snapshot per week between 2013-10-18 and 2014-09-19
17:30:45.292 INFO  - -> Keep one snapshot per month between 2009-10-23 and 2013-10-18
17:30:45.294 INFO  - -> Delete data prior to: 2009-10-23
17:30:45.302 INFO  - -> Clean MyJavaProject [id=160]
17:30:45.309 INFO  - Clean snapshot 5368
 
BUILD SUCCESSFUL

And of course, browsing to your Sonar instance should reveal some good information about your project.

Summary

In this post, I discussed how to configure Gradle to run a Sonar analysis on your code, and post the results back to your SonarQube instance. In a future post, I’ll look at more of the SonarQube analysis results, and talk about common fixes for various problems.