Manually Installing Pip and Wheel

I recently had an installation of Debian Wheezy on an an ARMv5 device that needed some new Python packages installed. Having performed some Python development in the past, I wanted to use Pip (a Python package manager).

I also needed to install some Python Wheels. Wheels are an interesting addition to the Python lifecycle. Since the packages are already pre-built, there is no worry about the whole build phase when installing a Python package – a phase that could really cause you to have a bad day if you don’t have a good build environment already installed.

Unfortunately, sometimes installing packages isn’t as straightforward as I would hope it would be – sometimes it is on those good days when the stars are aligned correctly, and your lucky rabbit’s foot is tuned up. But other times, it requires a little bit of persistence and tweaking until you get it right.

Installing from Repos

At first, I installed Pip from the repositories. I ran a quick search to find the correct package:

sudo apt-cache search pip | grep python

Installation was via a typical apt-get command:

sudo apt-get install python-pip

Afterwards, I used Pip itself to install the required Wheel package:

sudo pip install wheel

As a test, I tried to install a recent setuptools wheel package into the current directory, but was met with an error message:

pip wheel --wheel-dir=./ setuptools
pip: error: No command by the name pip wheel
  (maybe you meant "pip help --wheel-dir=./ setuptools")

Hmm… a quick check on the version confirmed it:

pip --version
pip 1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

The wheel command needs a much newer version of Pip. I did a quick apt-get update and checked the repos again:

sudo apt-get update
sudo apt-cache madison python-pip
python-pip |      1.1-3 | http://cdn.debian.net/debian/ wheezy/main armel Packages

Nope, looks like version 1.1 is as good as it’s going to get from the repos. Time for plan B.

Manual Installation

To prevent problems with having multiple binaries, I purged the existing Pip binary:

sudo apt-get purge python-pip
Reading package lists... Done
Building dependency tree        
Reading state information... Done
The following packages will be REMOVED:
  python-pip*
0 upgraded, 0 newly installed, 1 to remove and 75 not upgraded.
After this operation, 468 kB disk space will be freed.
Do you want to continue [Y/n]? Y
(Reading database ... 30164 files and directories currently installed.)
Removing python-pip ...
Processing triggers for man-db ...

After reading the documentation, the simple way to install was to grab the get-pip.py Python script directly from Pip’s bootstrap site using wget. It requires an HTTPS connection however, so it helps to make sure you set the secure protocol switch:

wget https://bootstrap.pypa.io/get-pip.py --secure-protocol=auto
--2014-09-26 23:56:33--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 199.27.79.175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|199.27.79.175|:443... connected.
ERROR: The certificate of `bootstrap.pypa.io' is not trusted.
ERROR: The certificate of `bootstrap.pypa.io' hasn't got a known issuer.

Looks like my root certificates were out of date. The quickest fix (if you are really worried about ghosts, spiders, EMPs and security) was to update the ca-certificates package:

sudo apt-get install ca-certificates
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  ca-certificates
0 upgraded, 1 newly installed, 0 to remove and 75 not upgraded.
Need to get 185 kB of archives.
After this operation, 337 kB of additional disk space will be used.
Get:1 http://cdn.debian.net/debian/ wheezy/main ca-certificates all 20130119 [185 kB]
Fetched 185 kB in 0s (437 kB/s)     
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package ca-certificates.
(Reading database ... 29993 files and directories currently installed.)
Unpacking ca-certificates (from .../ca-certificates_20130119_all.deb) ...
Processing triggers for man-db ...
Setting up ca-certificates (20130119) ...
Processing triggers for ca-certificates ...
Updating certificates in /etc/ssl/certs... 158 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.

Sure enough, the wget worked without problems:

wget https://bootstrap.pypa.io/get-pip.py --secure-protocol=auto
--2014-09-27 00:02:03--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 23.235.47.175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|23.235.47.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1340903 (1.3M) [text/x-python]
Saving to: `get-pip.py'
 
100%[================================================================>] 1,340,903   1.46M/s   in 0.9s    
 
2014-09-27 00:02:08 (1.46 MB/s) - `get-pip.py' saved [1340903/1340903]

After that, I finally installed Pip using the Python script:

sudo python ./get-pip.py
Downloading/unpacking pip
  Downloading pip-1.5.6-py2.py3-none-any.whl (1.0MB): 1.0MB downloaded
Installing collected packages: pip
Successfully installed pip
Cleaning up...

Okay, then I checked Pip’s version:

pip --version
pip 1.5.6 from /usr/local/lib/python2.7/dist-packages (python 2.7)

Looks like Pip was ready to go. A quick test to install the setuptools wheel to the current directory:

pip wheel --wheel-dir=./ setuptools
Downloading/unpacking setuptools
  Downloading setuptools-6.0-py2.py3-none-any.whl (533kB): 533kB downloaded
  Saved ./setuptools-6.0-py2.py3-none-any.whl
Cleaning up...

Now we’re cooking with gas!

Summary

Sometimes what should be simple package management turns out to be much more complicated. It pays to keep digging a little – usually the sources of the problems are due to outdated packages or unmet dependencies.

Deer Detection Diversion 1 – Detecting Day and Night

In a previous post, I wrote about how I wanted to use Machine Learning to detect when a deer is present in my back yard, so that I can make sure that no deer-built weapons of mass destruction are constructed in my yard (at least not on my watch!). After initially using a Microsoft 360 Kinect for my first data collection phase, I ran into a problem: deer (and many other animals) are actually nocturnal. The Kinect camera, while easy to use, doesn’t have a sensitive enough IR camera to see in the darkness of my backyard, even with the help of some nifty IR emitters.

To solve the problem of night vision, I wrote about how I set up a Raspberry Pi with a NOIR camera to keep track of the murderous deer in the yard both day and night. Here’s a picture of the new rig:

rpi_ir_camera_rig

My NOIR camera and Raspberry Pi. Plus my super low cost NOIR case – an SD  card holder.

The Challenge

While the camera and IR emitters work great for capturing pictures, I ran into a timing problem. The NOIR camera requires very different settings between day-time and night-time exposures.

During the day time, I can use the camera’s automatic white balance feature, along with an ISO of 100. However, at night I need to tweak the camera to use a very specific white balance, and adjust the shutter speed so that it stays open for nearly 3 seconds.

The main challenge comes when it is time to swap from day-mode to night-mode automatically. If I leave the NOIR camera on night mode for too long into the sunrise, I get images like this:

noir_night_dawn

Whoa! Is it a nuclear explosion with a dreaded EMP!? Oh… no, it’s just dawn.

Right now, I set the camera on a schedule. Using a small Python script, I get the camera to take a set number of pictures before stopping, and then swap settings from day to night, or night to day. This works okay, but there is one glaring problem – the sun rises and sets at a different time every day! Pesky sun!

The Solution

While I could solve the problem by hooking into a web service that tells me when the sun is about to rise or set, I think a more interesting solution is to have the Python script look at each exposure, and swap the day and night settings by itself.

Note that this sounds like I’m proposing some sort of self aware program… but I’m not talking about Skynet levels of intelligence – I don’t need to be avoiding murderous deer and a runaway, self-aware, weakly god-like super-intelligent Raspberry Pi. I’m thinking more along the lines of using color histograms. Nice, simple, and (usually) non-deadly color histograms.

Too Bright or Too Dark?

The problem of when to transition between camera modes boils down to sampling an image and calculating the average pixel intensity. When it’s dark out, most of the image will become very dim, resulting in a color histogram that is close to black. When it becomes light outside, the opposite is true – the color histogram will be close to white.

So, how does one actually compute a color histogram? It’s easy! Usually, images are stored (or can be transformed into) an RGB format where each pixel has 3 color components: R (red), G (green), and B (blue). Each of these components has an intensity value that will usually take on a value between 0 and 255. For example, if there was a pure red pixel, its color value might be R = 255, G = 0 and B = 0. Pure black is R = 0, G = 0, and B = 0. Pure white is R = 255, G = 255, and B = 255. A nice brown deer color would be R = 130, G = 88, and B = 64.

To compute a color histogram, the program needs to loop through each pixel in an image and read the red (R), green (G) and blue (B) intensities, summing up the totals for each intensity level. The result is something like this:

Intensity |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
 0        | 0        | 0          | 0          |
 1        | 0        | 0          | 0          |
 2        | 0        | 0          | 0          |
 3        | 0        | 0          | 0          |
 ...      | ...      | ...        | ...        |
 237      | 8101     | 5949       | 5940       |
 238      | 15527    | 8518       | 8842       |
 239      | 214433   | 13467      | 16682      |
 240      | 24592    | 24301      | 208580     |
 241      | 38842    | 50694      | 59539      |
 242      | 7938     | 183090     | 14318      |
 243      | 6045     | 750        | 5157       |
 244      | 2121     | 8          | 3872       |
 245      | 1550     | 0          | 2063       |
 246      | 779      | 0          | 1280       |
 247      | 368      | 0          | 1022       |
 248      | 311      | 0          | 721        |
 249      | 168      | 0          | 451        |
 250      | 132      | 0          | 350        |
 251      | 84       | 0          | 257        |
 252      | 80       | 0          | 207        |
 253      | 40       | 0          | 149        |
 254      | 28       | 0          | 125        |
 255      | 73       | 0          | 469        |

This is actual histogram from my nuclear dawn picture I showed above. I’ve truncated the data above and cut out intensities 4 – 236. Interpreting the chart is fairly straightforward. For example, there are 183,090 pixels that have a G value of 242, meaning that there is a lot of high green values. Clustered around the 240 intensity level are the bulk of the RGB values, which I would expect if the picture has a lot of white in it (which it does).

Actually accessing the RGB data in a program is easy – it turns out the OpenCV project has a Python library called cv2 that makes this possible:

import cv2
image = cv2.imread(filename)
blue = cv2.calcHist([image], [0], None, [256], [0, 256])
green = cv2.calcHist([image], [1], None, [256], [0, 256])
red = cv2.calcHist([image], [2], None, [256], [0, 256])

This produces the histogram information for the R, G, and B components of the image. What is more useful is to plot the histograms out to give you a feel for where the main bulk of the color intensities are located. MatPlotLib is the Python package to use. Here is what the histogram data look like:

color_histogram_nuclear_dawn

The color histogram for the nuclear dawn picture. Notice how the red, green and blue components are all really high and clustered together! This is important!

Now, let’s compare it to another picture. This is one where the camera was running in day mode, and it started to get dark out. Here is the picture:

day_night_transition

The world is going dark! Oh noes! What happened to the sun!?

Now let’s take a look at the histogram data:

color_histogram_darkness

This is the histogram for the picture where the sun is burning out! Ahem… I mean where the sun is setting… we probably have a few billion years before it actually burns out.

Notice that the histogram data on both images tends to polarize to one end of the spectrum or another – for the nearly white picture, the bulk of the components are near 255. For the dark image, the bulk are near zero.

I can use this to my advantage in the Python program. All I have to do is see if the intensity values for each color component are near 0 or 255. However, for this to work, I need to calculate the weighted mean of each of the R, G, and B components, and compare them to some cutoff values.

The weighted mean calculation works similar to the usual mean calculation. The difference is that instead of just summing up all of the counts for say the R component, I multiply the count by the intensity value, perform the sum, and then divide by the total number of pixels we have. So, for the R component, the calculation becomes:

R_weighted_sum = ((0 * R[0]) + (1 * R[1]) + ... (255 * R[255])) / sum(R)

I just repeat that weighted sum calculation for the G, and B components. For example, the weighted values for the nuclear dawn photo is as follows:

          |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
Wgt-Mean  | 238      | 237        | 240        |

Putting It All Together

All that I needed now was to get my Python camera implementation to check for histogram data every few snapshots, and calculate the weighted sum of intensities. To get it to transition into day mode, I look for the average intensities of the R, G and B values to be greater than 230. For the night mode, I look for the average intensities to be less than 30. When it detects either of those situations, it will adjust the camera settings accordingly.

As an example, I created a time-lapse view from 6:00 am to 6:12 am. The program bounces back and forth a little between modes until it becomes bright enough in day mode to use the automatic settings (note that the animation loops!). I can tweak this a little by using slightly different cutoffs for the pixel intensities:

transition

Time-lapse view of the transition from night mode to day mode. The flicker back and forth is because it’s still too dark for day mode. As the sun gets brighter, the program stabilizes in day mode.

The full source code for the program is available in my GitHub repository. Check it out and feel free to experiment with it!

Summary

Using histogram data to swap camera modes is very easy. I showed how to compute the histogram data, and calculate the weighted mean of the intensity values. With both of these tools in hand, it’s easy to determine whether the camera should switch to night mode or day mode based on the light values! As long as the deer doesn’t figure out how to blow up the sun, I can leave my Raspberry Pi camera program running until I run out of disk space. This will collect good evidence if the deer manages to murder me, or, more likely, eats all of my vegetables.

Check back soon when I will (finally) discuss the Machine Learning component of the project.