Android and Bamboo – Testing Using the Android Emulator on a CI Server

In a previous post, I discussed how to set up Atlassian Bamboo to build unit tests as part of an Android project. In that post, I went through the steps needed to install the Android SDK on a Bamboo build server, and configure a Bamboo build project to build and test the codebase.

One of the assumptions that I made in the previous post was that all of the unit tests were written with a package called Robolectric (see my post on Android Unit Testing with Robolectric for details on how to set up and use it). Why use Robolectric? Well, it allows you to test your project without having to run the tests on an Android emulator. As you may already know, the emulator – while good – can be slow. However, there are many times when you need to test on an actual Android device. In this post, I will discuss how to run an emulator on your CI server, and configure Bamboo to run Android tests directly in the emulated device.

Assumptions

In this post, I’m going to make several assumptions:

  1. Your Bamboo server is running a Linux OS.
  2. You have already installed the SDK on the Bamboo server, and that it has the components you need to build and support your Android application. If you don’t already have this set up, see my previous post on how to do this.
  3. You are familiar enough with Bamboo that you know how to configure a project, and add several tasks to it.
  4. You have access to a shell on the build server so that you can define a few things that Bamboo needs.
  5. You want to run tests that actually require an emulator (or device) to run. Tests such as ActivityInstrumentationTestCase2 and ProviderTestCase2 are good examples of tests that need to be run on an emulator or device.

The Solution

Implementing this solution involves the following steps:

  1. Define an emulated device.
  2. Start a headless emulator in the background, and monitor it.
  3. In Bamboo, add a task to run the tests on the emulator.

Define an Emulated Device

You need to shell into your build server now. Once there, you need to define an emulator with the configuration your application requires. In my case, I needed an emulator that supported API level 19 (KitKat). In the example below, I’m also using the x86 image for the emulator. I called my new emulator android_x86. Make sure you switch to the user that has access to the Android SDK that you installed.

android create avd --force -n android_x86 -t android-19 --abi x86

Side-note: why use the x86 image? If you are have either an Intel or AMD processor on Linux, the x86 image can be accelerated as long as you have KVM installed. See my post on Android Emulator VM Acceleration on AMD for details on how to install KVM.

Starting and Monitoring a Headless Emulator

For this task, we will create a start/stop script for an Android emulator, and install and use the Linux program monit to spin up the emulator and make sure it continues to run. The monit program is a great monitoring program that will be able to see if an emulator is running, and if not, start one up in headless mode. Plus, it has a web interface that will let us start, stop, and monitor the status of the running Android emulator from within a browser.

An Emulator Start/Stop Script

The first thing to do is to create a script that will be responsible for starting and stopping the emulator in headless mode. What is headless mode? Usually a build server does not have a monitor attached to it, and thus anything that tries to open a window on the display will error out. Operating the emulator in headless mode will prevent that from happening, since it won’t try and display an emulator window. A few blogs out on the net have good instructions on how to do this (for example, Paul Estrada’s post on defining a headless emulator for Travis CI).

The script below acts as a simple start/stop script. I named the script android-emulator-control.sh and put it in /opt/bin for easy access. You will need to edit ANDROID_HOME, ANDROID_SDK_HOME, and the EMULATOR_PORT depending on where your Android environment is configured, and what port you want to run it on. You’ll also need to edit the AVD_NAME to match the name of the AVD you created in the previous step. The script will start up an emulator on the designated port (in this case port 5556). It will also create a process identification file at /tmp/android-emulator-5556.pid and a log file at /tmp/android-emulator-5556.log.

#!/bin/bash
export ANDROID_HOME=/opt/android/android-sdk-linux
export ANDROID_SDK_HOME=/opt/android/android-sdk-linux
EMULATOR_PORT=5556
PID_FILE="/tmp/android-emulator-$EMULATOR_PORT.pid"
LOG_FILE="/tmp/android-emulator-$EMULATOR_PORT.log"
EXEC_FILE="$ANDROID_HOME/tools/emulator"
AVD_NAME="android_x86"
 
case $1 in
    start)
        $EXEC_FILE -avd $AVD_NAME -port $EMULATOR_PORT -no-skin -no-audio -no-window \
            -qemu -m 512 -enable-kvm > $LOG_FILE 2>&1 & echo $! > $PID_FILE
        ;;
    stop)
        kill `cat $PID_FILE`
        ;;
    *)
        echo "usage: android-emulator-control.sh {start|stop}" ;;
esac
exit 0

Once you have created the script, you will need to make it executable with:

sudo chmod +x /opt/bin/android-emulator-control.sh

With the script executable, to start the emulator, just do a /opt/bin/android-emulator-control.sh start. To stop it is /opt/bin/android-emulator-control.sh stop.

Installing monit

Next is to install monit if it is not already installed. On a Ubuntu Server, this can be accomplished with:

sudo apt-get install monit

Configuring monit

The next step is to get the monit daemon configured. The general configuration file is located at /etc/monit/monitrc. To edit it, you will need to:

sudo vi /etc/monit/monitrc

You will need to enable or add the following lines in order to make it accessible on the web. Replace the IP address 10.0.0.44 with your server’s IP address, and the 10.0.0.0/24 with the subnet that you want to be able to access the monitor program (e.g. to give everyone in the 192.168.0 subnet access, make it 192.168.0.0/24):

 set httpd port 2812 and
    use address 10.0.0.44
    allow admin:monit
    allow 10.0.0.0/24

Save the file and exit. Next, you will need to create a configuration file for monit that will monitor the Android emulator. To do that, create a new file called /etc/monit/conf.d/android-emulator and add the following code:

check process emulator-5556 with pidfile /tmp/android-emulator-5556.pid
   start program = "/opt/bin/android-emulator-control.sh start"
      as uid bamboo and gid bamboo
   stop program = "/opt/bin/android-emulator-control.sh stop"
      as uid bamboo and gid bamboo
   if failed host localhost port 5556 then restart

Save the file and exit. This configuration file will tell monit to monitor the emulator PID, and restart the program if it finds it is no longer running. It also is configured to run the process as user bamboo with the group bamboo. This is because the bamboo user actually owns the emulator SDK files, and is the only user who is able to run the emulator and other Android commands.

Checking that monit is Running

Okay, now for the moment of truth. If you have configured monit correctly, it is time to start it up and see if it runs the emulator correctly. To start up monit:

sudo /etc/init.d/monit stop
sudo /etc/init.d/monit start

When it is restarted, aim your browser at the port 2812 on your CI server. You should be prompted for a username and password. Use the username and password that you defined in the monitrc file. You should be greeted with a screen that displays the status of your system, and the status of the emulator:

monit-main-screen

If you click on emulator-5556 you will be taken to a screen with more information about the process:

monit-emulator-options

As you can see, there are options here to start and stop the emulator. This is really useful if Bamboo says that there was an error with the emulator. You can use this web screen to monitor its status when you are troubleshooting.

Configuring Bamboo to Test the Project

Okay, the hard part is done. The final step is to configure the Bamboo project to execute the tests. Create a new Script task after code checkout has been completed and call it Run Tests on Emulator. You will need to pass the applicable ANDROID_HOME path. Simply create a script task, and add the following command to it:

./gradlew connectedAndroidTest

That’s all there is to it. Here is a screenshot of my Bamboo task that does this:

bamboo-script-config-android

You will also need a JUnit Parser task in order to pick up the results of the tests. Add this as another task right after the test task. Call it Parse Test Results. You will need to specify the path to the generated XML reports. For Android projects, they are placed in the app/build directory under **/app/build/outputs/androidTest-results/connected/*.xml:

junit-parser-config

Run the Tests!

With everything set up, you should now be able to run the tests. If everything works correctly, the results of your run should be available in Bamboo:

bamboo-successful-android-build

Here you can see my 3 tests ran correctly. These were tests that required instantiating an Activity and checking properties of various objects.

Troubleshooting

Bamboo Throws Errors

The most common error is that when the tests are run, the emulator isn’t up and running. If you look at the gradle output in Bamboo, you will see messages similar to the following:

:app:connectedAndroidTestDebug FAILED
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Execution failed for task ':app:connectedAndroidTestDebug'.
> com.android.builder.testing.api.DeviceException: java.lang.RuntimeException: No connected devices!

The solution here is to make sure the emulator is actually up and running. If the above script is not working for you, you may need to take another approach to running the emulator.

The Emulator Won’t Run

There are many reasons why the emulator will not run from the simple start/stop script outlined above. For most of them, setting the path to the SDK is key, and making sure the AVD images are where the emulator command expects to see them. Your best bet is to look at the log file that is generated when the start/stop script is called. In my example above, it will be at /tmp/android-emulator-5556.log. It will usually give you enough error information to help you figure out what went wrong. Also, be sure that you have created the emulator image with the android command with the user that has access to your Android SDK.

Multiple Build Agents

While the above works well if you have a single build agent, things get more complicated if you have multiple build agents running on the same machine. If more than one build agent tries to access the emulator at a time, you will probably get some unexpected results (crashes or failures). My suggestion would be to have several different remote agents. On each remote agent machine, you can have an emulator running using the process I described above.

Alternatively, define only a single agent with the capability to run Android tests. This will mean that on a busy server you will experience a backlog of jobs that need to be built, but at least you will not cause collisions with the emulator and different build agents. As a compromise, creating a single nightly run that executes these type of emulator tests on your development branch may be a good alternative to running them every time a branch is created or updated.

Wrapping Up

In this post, I discussed a solution for running Android tests on a Bamboo CI server. It involves running an emulator in the background so that the build agent can connect to it when it needs to run tests. Other CI servers have interesting plugins that help with the automated configuration and provisioning of emulators on the fly (for example, Jenkins has an Android Emulator Plugin that helps out with this task). The solution I describe above used a simple command called monit to run an emulator in the background.

Hopefully you have been able to implement the steps above, and have Bamboo building your latest Android project! Feel free to asks questions or leave comments in the comment section below.

Continuous Integration with Android and Bamboo

In a previous post, I spoke about how to use Robolectric to unit test Android applications without having to resort to the emulator. In this post, I’ll detail setting up Atlassian Bamboo to run continuous integration of an Android application.

What is Bamboo?

For those of you who don’t know, Atlassian Bamboo is a Continuous Integration system. Many people have probably heard of Confluence (think internal wiki and collaboration software), Stash (think BitBucket) or JIRA (issue tracking software). These three systems are widely used in Enterprise software development shops and many businesses in general.

To me, Bamboo is a vital key in the software development process. Combining Bamboo with JIRA and Stash provides some really great features, such as creating task branches directly from your issue tracking software, seeing what issues were resolved with a commit, and being able to track released versions with all of their fixes detailed in a nice changelog.

The Problem

When performing Android development, I needed to get Bamboo building my projects. Doing so involved two separate pieces:

  1. Installing the Android SDK so that Bamboo had access to it.
  2. Configuring Bamboo to build the actual project.

Note that I’m assuming that you already have a JDK installed on the server, and that you’ve already installed Bamboo.

Installing the Android SDK

This task was fairly straight-forward. First, I had to download the actual Android SDK from Google’s website. This meant visiting the Android SDK website, and then clicking on View all downloads and sizes. I just wanted the SDK tools, not the ADT bundle, so I selected the android-sdk_r23.0.2-linux.tgz package which was about 140 MB in size (note that when you read this, the SDK may be several versions more advanced – your mileage, therefore, may vary).

Once downloaded, I copied it up to my server, and expanded it in a place where all users of the server could access it:

sudo mkdir /opt/android
cd /opt/android
sudo tar xvzf ~thomas/android-sdk_r23.0.2-linux.tgz

This expanded the SDK to the following directory:

/opt/android/android-sdk-linux

I then fired up the SDK manager to install the necessary components. First though, I had to log out, and then SSH with X-11 forwarding turned on so that I could see the GUI it provides (note that you can do everything without the GUI using the --no-ui option for the android update sdk command):

ssh -X 10.0.0.60

Next, I actually ran the android tool as root:

sudo /opt/android/android-sdk-linux/tools/android

However, this resulted in an error message:

Exception in thread "main" java.lang.UnsatisfiedLinkError: no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path, java.library.path or the jar file
    at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
    at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
    at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
    at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
    at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
    at org.eclipse.swt.widgets.Display.(Unknown Source)
    at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:402)
    at com.android.sdkmanager.Main.doAction(Main.java:390)
    at com.android.sdkmanager.Main.run(Main.java:150)
    at com.android.sdkmanager.Main.main(Main.java:116)

The quickest way to solve this problem (as discussed here), was to install the libswt-gtk-3-java package:

sudo apt-get update
sudo apt-get install sudo apt-get install libswt-gtk-3-java

I then re-ran the SDK manager:

sudo /opt/android/android-sdk-linux/tools/android

And got the normal SDK manager window up. I then proceeded to install the components I needed as usual:

android-sdk-remote

Configuring Bamboo

The next step was to configure Bamboo to actually perform the build. I went ahead and created a new Plan for my code repository. When configuring the tasks, I left the default Stage in place. Under the default stage, I left the default Source Code Checkout task in place, but added a new task to perform the build. Because I was using Gradle, I created a Script task:

bamboo-script-task

The script task required only two pieces of information. For the actual script, I used the Gradle wrapper that I had checked into source code control, and used the build target (./gradlew build). In addition to that, I had to tell the script where the Android SDK was located. For that, I created an environment variable called ANDROID_HOME and set it to where the SDK was located. Here is a screenshot of the configured task:

bamboo-gradle-build

When I first ran the build, I was surprised that no test results were being reported, even though the build target in the Gradle wrapper runs the actual unit tests. I realized I needed to use a JUnit Parser task to parse the test results. All that was needed for that was to specify the directory where the results of the unit tests would be located. I did that with a simple ANT-style path:

**/test-results/*.xml

Here is a screenshot of the configured JUnit Parser task:

junit-parser-test-results

With that complete, I triggered a manual build, and checked the test results tab to make sure everything ran:

bamboo-test-results

83 tests run, all successful. Excellent!

Wrapping Up

As you can see, it is fairly straightforward to get simple unit testing going on Bamboo with the Android SDK. This assumes of course, that you don’t actually need the emulator, and are using a package such as Robolectric. In a future post, I’ll talk about some more of the challenges of developing a CI strategy with the Android SDK, and how to solve them.