Booting a Raspberry Pi From a 16 MB SD Card

Raspberry Pis make great little servers. With low power consumption, they make great devices if you need cheap web services running 24/7. One aspect I dislike about them however, is that they need to boot using an SD card. Given the limited number of write cycles that most SD cards have, I hate having to run anything more than the bootloader off of one.

While there are many tutorials that talk about how you can clone the SD card to a USB drive, I wanted to do something a little different. Why waste a 4 GB SD card? I have lots of smaller SD cards kicking around. In this post, I’ll discuss how I made my Raspberry Pi boot from a 16 MB SD card and use a USB hard drive for the root filesystem.

Running the Pi from a USB Hard Drive

Most tutorials have you simply clone your SD card to a USB hard drive, and then edit the cmdline.txt file to point to the USB hard drive when booting the main OS. This solves the problem of moving the root filesystem to a more robust storage medium, but ties up a 4 GB SD card that I could be using elsewhere. I know this isn’t too big of a problem, but it does mean buying a bunch of 4 GB SD cards for all my Raspberry Pis.

On a related note, I have a number of digital cameras that I have purchased over the years, and most of them came with a 16 MB SD card. Back in the day, 16 MB was a lot of storage space, but given the megapixel counts that come with most cameras, 16 MB can usually only store a few pictures. In my case, they sit in a stack in my desk drawer, gathering dusk and sulking.

This got me thinking: I’d really love to use the 16 MB card in the Raspberry Pi (plus a hard drive), and use the 4 GB card in my cameras.

Update January 20, 2015: Mike Redrobe discussed on his blog how to do this in May 2014, with great instructions on how to do this under Windows. Also, this blog noted how to do it in 2012, but without the step-by-step breakdown. Many other sources (like this blog, and this blog) outline how to modify the boot partition to mount the root filesystem from a USB drive.

What is Where

I took one of my 4 GB SD cards that had Raspbian on it, and examined how the card was formatted. There is a small MS-DOS partition used for the bootloader, and the rest of the card is used for the root filesystem. The MS-DOS partition is what interests me, because this is what the Raspberry Pi needs to see in order to bootstrap the OS. Here’s the contents:

# ls -lh /boot
total 9.7M
-rwxr-xr-x 1 root root  18K Dec 21 10:53 bootcode.bin
-rwxr-xr-x 1 root root  120 Dec 21 11:14 cmdline.txt
-rwxr-xr-x 1 root root 1.3K Dec 21 11:14 config.txt
-rwxr-xr-x 1 root root 2.3K Dec 21 10:53 fixup_cd.dat
-rwxr-xr-x 1 root root 6.0K Dec 21 10:53 fixup.dat
-rwxr-xr-x 1 root root 9.0K Dec 21 10:53 fixup_x.dat
-rwxr-xr-x 1 root root  137 Dec 24 12:29 issue.txt
-rwxr-xr-x 1 root root 3.2M Dec 21 10:53 kernel.img
-rwxr-xr-x 1 root root  19K Sep 25  2013 LICENSE.oracle
-rwxr-xr-x 1 root root 538K Dec 21 10:53 start_cd.elf
-rwxr-xr-x 1 root root 2.6M Dec 21 10:53 start.elf
-rwxr-xr-x 1 root root 3.5M Dec 21 10:53 start_x.elf

Most of those files look pretty small… hmm… I wonder how much disk space is actually used:

# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc1        56M  9.7M   47M  18% /mnt/raspbianboot

So on the 60 MB MS-DOS boot partition, 9.7 MB worth of files are used for the bootloader. This is perfect! I can fit those files on a 16 MB SD card!

Preparing the 16 MB SD Card

Step 1 was to format the 16 MB SD card as needed in order to boot Raspbian. So, I stuck the card in the computer and took a look at the partitions on it (under Ubuntu, the device registered as /dev/sdc).

Disclaimer: I use fdisk to delete partitions on the card. If you accidentally specify the wrong device, you can destroy your hard disk partition and make your computer unbootable. These instructions are for educational purposes only. Running any of the commands below is at your own risk!

# sudo fdisk -l /dev/sdc
 
Disk /dev/sdc: 16 MB, 16056320 bytes
2 heads, 32 sectors/track, 490 cylinders, total 31360 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              57       31359       15651+   1  FAT12

Okay, so this is the correct device – 16 MB total storage. It’s currently formatted as FAT12. For the Raspberry Pi to boot, we need a FAT32 filesystem instead. So, I started up fdisk and deleted the primary partition:

# sudo fdisk /dev/sdc
 
Command (m for help): d
Selected partition 1

Next, I created a new primary partition on the card:

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-31359, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-31359, default 31359): 
Using default value 31359

Next, I changed the partition to W95 FAT32 (LBA). This is type ID c:

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): c

Finally, I needed to mark the new primary partition as bootable:

Command (m for help): a
Partition number (1-4): 1

With that done, I saved the partition information, and quit fdisk:

Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

Note: if you receive messages about the system retaining the old partition information after you create the new primary, simply remove the card from the card reader, and then re-insert it.

Finally, I created the filesystem on the new partition:

# sudo mkfs -t vfat /dev/sdc1
mkfs.fat 3.0.26 (2014-03-07)

Copying the Boot Files

The next step was to copy the boot files from the original Raspbian 4 GB card to the 16 MB card. I removed the 16 MB card from my card reader, and popped in the Raspbian 4 GB SD card. I created a mount point for it, and mounted the boot partition. Note that the first partition on the card (again registered as /dev/sdc) contains the boot partition. So /dev/sdc1 is the boot partition, while /dev/sdc2 contains the root filesystem:

# sudo mkdir /mnt/raspbianboot
# sudo mount /dev/sdc1 /mnt/raspbianboot

I needed a temporary location to store the files, so I copied them to /tmp:

# sudo cp -r /mnt/raspbianboot /tmp

So far so good. Now to put them onto the 16 MB card. I unmounted the boot partition:

# sudo umount /mnt/raspbianboot

I popped out the Raspbian 4 GB SD card and put in the 16 MB card. To keep things clear, I created a new mount point for it… just so I didn’t confuse myself with what I was mounting. Again, when I took out the 4 GB card and put in the 16 MB card, it registered as /dev/sdc:

# sudo mkdir /mnt/raspbianboot16
# sudo mount /dev/sdc1 /mnt/raspbianboot16

Next, I copied the files from /tmp to the 16 MB SD card:

# sudo cp -r /tmp/raspbianboot/* /mnt/raspbianboot16

The final thing was to modify the boot loader on the 16 MB card to boot from the USB hard drive instead of trying to boot on the second partition of the SD card (which would be bad, since there isn’t one!):

# sudo vi /mnt/raspbianboot16/cmdline.txt

I changed:

root=/dev/mmcblk0p2

To the following:

root=/dev/sda1

I also added two additional flags to the file at the end of the line:

bootdelay rootdelay

I saved the file and unmounted the drive:

# sudo sync
# sudo umount /mnt/raspbianboot16

Cloning Raspbian Wheezy to the USB Hard Drive

The last piece of the puzzle is to clone the other Raspbian root filesystem on the SD card to the USB hard drive. I have a 320 GB USB drive that I use for things like my Deer Detector. First up, I put the Raspbian 4 GB SD card back into my card reader, which registered as /dev/sdc. Next, I plugged in the new USB hard drive. In my machine, the new drive shows up as /dev/sdd. I copied the root partition from the Raspbian 4 GB SD card to the new disk. Note that since I want the root partition, it’s /dev/sdc2 that I want to copy from (remember /dev/sdc1 was the boot partition):

# sudo dd if=/dev/sdc2 of=/dev/sdd bs=4M

One final housekeeping point remained. The Raspbian root filesystem has an fstab that tries to mount / from the SD card. I needed to change that. So, I mounted the new USB hard drive under /mnt/raspbianroot and modified the file:

# sudo mkdir /mnt/raspbianroot
# sudo mount /dev/sdd1 /mnt/raspbianroot
# sudo vi /mnt/raspbianroot/etc/fstab

I needed to change the following line:

/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1

To this:

/dev/sda1  /               ext4    defaults,noatime  0       1

With that done, I unmounted the USB drive, and plugged both the 16 MB SD card and the USB hard drive into the Raspberry Pi (to power the drive, I had to use a powered USB hub, but you get the idea). The Raspberry Pi booted perfectly, and I verified that it was running from the USB hard drive by checking that /dev/sda1 was mounted at /.

Note: the new partition on the USB hard drive will be exactly the size as the partition on the 4 GB SD card. Since I wanted to extend the partition to take up the full drive, I ran fdisk and printed out the current partition information, and made note of the starting block. Then, I deleted the partition, and recreated it using the same starting block. I made sure the partition spanned the entire drive. I wrote the table out, rebooted, and then ran resize2fs to resize the filesystem.

Wrapping Up

Making the Raspberry Pi boot from a 16 MB SD card was pretty easy. Essentially it involves cloning the boot partition to the smaller SD card and the root partition to some other USB storage medium. As long as the boot partition continues to be smaller than 16 MB, this trick will continue to work.

Deer Detection Diversion 1 – Detecting Day and Night

In a previous post, I wrote about how I wanted to use Machine Learning to detect when a deer is present in my back yard, so that I can make sure that no deer-built weapons of mass destruction are constructed in my yard (at least not on my watch!). After initially using a Microsoft 360 Kinect for my first data collection phase, I ran into a problem: deer (and many other animals) are actually nocturnal. The Kinect camera, while easy to use, doesn’t have a sensitive enough IR camera to see in the darkness of my backyard, even with the help of some nifty IR emitters.

To solve the problem of night vision, I wrote about how I set up a Raspberry Pi with a NOIR camera to keep track of the murderous deer in the yard both day and night. Here’s a picture of the new rig:

rpi_ir_camera_rig

My NOIR camera and Raspberry Pi. Plus my super low cost NOIR case – an SD  card holder.

The Challenge

While the camera and IR emitters work great for capturing pictures, I ran into a timing problem. The NOIR camera requires very different settings between day-time and night-time exposures.

During the day time, I can use the camera’s automatic white balance feature, along with an ISO of 100. However, at night I need to tweak the camera to use a very specific white balance, and adjust the shutter speed so that it stays open for nearly 3 seconds.

The main challenge comes when it is time to swap from day-mode to night-mode automatically. If I leave the NOIR camera on night mode for too long into the sunrise, I get images like this:

noir_night_dawn

Whoa! Is it a nuclear explosion with a dreaded EMP!? Oh… no, it’s just dawn.

Right now, I set the camera on a schedule. Using a small Python script, I get the camera to take a set number of pictures before stopping, and then swap settings from day to night, or night to day. This works okay, but there is one glaring problem – the sun rises and sets at a different time every day! Pesky sun!

The Solution

While I could solve the problem by hooking into a web service that tells me when the sun is about to rise or set, I think a more interesting solution is to have the Python script look at each exposure, and swap the day and night settings by itself.

Note that this sounds like I’m proposing some sort of self aware program… but I’m not talking about Skynet levels of intelligence – I don’t need to be avoiding murderous deer and a runaway, self-aware, weakly god-like super-intelligent Raspberry Pi. I’m thinking more along the lines of using color histograms. Nice, simple, and (usually) non-deadly color histograms.

Too Bright or Too Dark?

The problem of when to transition between camera modes boils down to sampling an image and calculating the average pixel intensity. When it’s dark out, most of the image will become very dim, resulting in a color histogram that is close to black. When it becomes light outside, the opposite is true – the color histogram will be close to white.

So, how does one actually compute a color histogram? It’s easy! Usually, images are stored (or can be transformed into) an RGB format where each pixel has 3 color components: R (red), G (green), and B (blue). Each of these components has an intensity value that will usually take on a value between 0 and 255. For example, if there was a pure red pixel, its color value might be R = 255, G = 0 and B = 0. Pure black is R = 0, G = 0, and B = 0. Pure white is R = 255, G = 255, and B = 255. A nice brown deer color would be R = 130, G = 88, and B = 64.

To compute a color histogram, the program needs to loop through each pixel in an image and read the red (R), green (G) and blue (B) intensities, summing up the totals for each intensity level. The result is something like this:

Intensity |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
 0        | 0        | 0          | 0          |
 1        | 0        | 0          | 0          |
 2        | 0        | 0          | 0          |
 3        | 0        | 0          | 0          |
 ...      | ...      | ...        | ...        |
 237      | 8101     | 5949       | 5940       |
 238      | 15527    | 8518       | 8842       |
 239      | 214433   | 13467      | 16682      |
 240      | 24592    | 24301      | 208580     |
 241      | 38842    | 50694      | 59539      |
 242      | 7938     | 183090     | 14318      |
 243      | 6045     | 750        | 5157       |
 244      | 2121     | 8          | 3872       |
 245      | 1550     | 0          | 2063       |
 246      | 779      | 0          | 1280       |
 247      | 368      | 0          | 1022       |
 248      | 311      | 0          | 721        |
 249      | 168      | 0          | 451        |
 250      | 132      | 0          | 350        |
 251      | 84       | 0          | 257        |
 252      | 80       | 0          | 207        |
 253      | 40       | 0          | 149        |
 254      | 28       | 0          | 125        |
 255      | 73       | 0          | 469        |

This is actual histogram from my nuclear dawn picture I showed above. I’ve truncated the data above and cut out intensities 4 – 236. Interpreting the chart is fairly straightforward. For example, there are 183,090 pixels that have a G value of 242, meaning that there is a lot of high green values. Clustered around the 240 intensity level are the bulk of the RGB values, which I would expect if the picture has a lot of white in it (which it does).

Actually accessing the RGB data in a program is easy – it turns out the OpenCV project has a Python library called cv2 that makes this possible:

import cv2
image = cv2.imread(filename)
blue = cv2.calcHist([image], [0], None, [256], [0, 256])
green = cv2.calcHist([image], [1], None, [256], [0, 256])
red = cv2.calcHist([image], [2], None, [256], [0, 256])

This produces the histogram information for the R, G, and B components of the image. What is more useful is to plot the histograms out to give you a feel for where the main bulk of the color intensities are located. MatPlotLib is the Python package to use. Here is what the histogram data look like:

color_histogram_nuclear_dawn

The color histogram for the nuclear dawn picture. Notice how the red, green and blue components are all really high and clustered together! This is important!

Now, let’s compare it to another picture. This is one where the camera was running in day mode, and it started to get dark out. Here is the picture:

day_night_transition

The world is going dark! Oh noes! What happened to the sun!?

Now let’s take a look at the histogram data:

color_histogram_darkness

This is the histogram for the picture where the sun is burning out! Ahem… I mean where the sun is setting… we probably have a few billion years before it actually burns out.

Notice that the histogram data on both images tends to polarize to one end of the spectrum or another – for the nearly white picture, the bulk of the components are near 255. For the dark image, the bulk are near zero.

I can use this to my advantage in the Python program. All I have to do is see if the intensity values for each color component are near 0 or 255. However, for this to work, I need to calculate the weighted mean of each of the R, G, and B components, and compare them to some cutoff values.

The weighted mean calculation works similar to the usual mean calculation. The difference is that instead of just summing up all of the counts for say the R component, I multiply the count by the intensity value, perform the sum, and then divide by the total number of pixels we have. So, for the R component, the calculation becomes:

R_weighted_sum = ((0 * R[0]) + (1 * R[1]) + ... (255 * R[255])) / sum(R)

I just repeat that weighted sum calculation for the G, and B components. For example, the weighted values for the nuclear dawn photo is as follows:

          |  Red     |    Green   |    Blue    |
----------+----------+------------+------------+
Wgt-Mean  | 238      | 237        | 240        |

Putting It All Together

All that I needed now was to get my Python camera implementation to check for histogram data every few snapshots, and calculate the weighted sum of intensities. To get it to transition into day mode, I look for the average intensities of the R, G and B values to be greater than 230. For the night mode, I look for the average intensities to be less than 30. When it detects either of those situations, it will adjust the camera settings accordingly.

As an example, I created a time-lapse view from 6:00 am to 6:12 am. The program bounces back and forth a little between modes until it becomes bright enough in day mode to use the automatic settings (note that the animation loops!). I can tweak this a little by using slightly different cutoffs for the pixel intensities:

transition

Time-lapse view of the transition from night mode to day mode. The flicker back and forth is because it’s still too dark for day mode. As the sun gets brighter, the program stabilizes in day mode.

The full source code for the program is available in my GitHub repository. Check it out and feel free to experiment with it!

Summary

Using histogram data to swap camera modes is very easy. I showed how to compute the histogram data, and calculate the weighted mean of the intensity values. With both of these tools in hand, it’s easy to determine whether the camera should switch to night mode or day mode based on the light values! As long as the deer doesn’t figure out how to blow up the sun, I can leave my Raspberry Pi camera program running until I run out of disk space. This will collect good evidence if the deer manages to murder me, or, more likely, eats all of my vegetables.

Check back soon when I will (finally) discuss the Machine Learning component of the project.