Android and Bamboo – Testing Using the Android Emulator on a CI Server

In a previous post, I discussed how to set up Atlassian Bamboo to build unit tests as part of an Android project. In that post, I went through the steps needed to install the Android SDK on a Bamboo build server, and configure a Bamboo build project to build and test the codebase.

One of the assumptions that I made in the previous post was that all of the unit tests were written with a package called Robolectric (see my post on Android Unit Testing with Robolectric for details on how to set up and use it). Why use Robolectric? Well, it allows you to test your project without having to run the tests on an Android emulator. As you may already know, the emulator – while good – can be slow. However, there are many times when you need to test on an actual Android device. In this post, I will discuss how to run an emulator on your CI server, and configure Bamboo to run Android tests directly in the emulated device.

Assumptions

In this post, I’m going to make several assumptions:

  1. Your Bamboo server is running a Linux OS.
  2. You have already installed the SDK on the Bamboo server, and that it has the components you need to build and support your Android application. If you don’t already have this set up, see my previous post on how to do this.
  3. You are familiar enough with Bamboo that you know how to configure a project, and add several tasks to it.
  4. You have access to a shell on the build server so that you can define a few things that Bamboo needs.
  5. You want to run tests that actually require an emulator (or device) to run. Tests such as ActivityInstrumentationTestCase2 and ProviderTestCase2 are good examples of tests that need to be run on an emulator or device.

The Solution

Implementing this solution involves the following steps:

  1. Define an emulated device.
  2. Start a headless emulator in the background, and monitor it.
  3. In Bamboo, add a task to run the tests on the emulator.

Define an Emulated Device

You need to shell into your build server now. Once there, you need to define an emulator with the configuration your application requires. In my case, I needed an emulator that supported API level 19 (KitKat). In the example below, I’m also using the x86 image for the emulator. I called my new emulator android_x86. Make sure you switch to the user that has access to the Android SDK that you installed.

android create avd --force -n android_x86 -t android-19 --abi x86

Side-note: why use the x86 image? If you are have either an Intel or AMD processor on Linux, the x86 image can be accelerated as long as you have KVM installed. See my post on Android Emulator VM Acceleration on AMD for details on how to install KVM.

Starting and Monitoring a Headless Emulator

For this task, we will create a start/stop script for an Android emulator, and install and use the Linux program monit to spin up the emulator and make sure it continues to run. The monit program is a great monitoring program that will be able to see if an emulator is running, and if not, start one up in headless mode. Plus, it has a web interface that will let us start, stop, and monitor the status of the running Android emulator from within a browser.

An Emulator Start/Stop Script

The first thing to do is to create a script that will be responsible for starting and stopping the emulator in headless mode. What is headless mode? Usually a build server does not have a monitor attached to it, and thus anything that tries to open a window on the display will error out. Operating the emulator in headless mode will prevent that from happening, since it won’t try and display an emulator window. A few blogs out on the net have good instructions on how to do this (for example, Paul Estrada’s post on defining a headless emulator for Travis CI).

The script below acts as a simple start/stop script. I named the script android-emulator-control.sh and put it in /opt/bin for easy access. You will need to edit ANDROID_HOME, ANDROID_SDK_HOME, and the EMULATOR_PORT depending on where your Android environment is configured, and what port you want to run it on. You’ll also need to edit the AVD_NAME to match the name of the AVD you created in the previous step. The script will start up an emulator on the designated port (in this case port 5556). It will also create a process identification file at /tmp/android-emulator-5556.pid and a log file at /tmp/android-emulator-5556.log.

#!/bin/bash
export ANDROID_HOME=/opt/android/android-sdk-linux
export ANDROID_SDK_HOME=/opt/android/android-sdk-linux
EMULATOR_PORT=5556
PID_FILE="/tmp/android-emulator-$EMULATOR_PORT.pid"
LOG_FILE="/tmp/android-emulator-$EMULATOR_PORT.log"
EXEC_FILE="$ANDROID_HOME/tools/emulator"
AVD_NAME="android_x86"
 
case $1 in
    start)
        $EXEC_FILE -avd $AVD_NAME -port $EMULATOR_PORT -no-skin -no-audio -no-window \
            -qemu -m 512 -enable-kvm > $LOG_FILE 2>&1 & echo $! > $PID_FILE
        ;;
    stop)
        kill `cat $PID_FILE`
        ;;
    *)
        echo "usage: android-emulator-control.sh {start|stop}" ;;
esac
exit 0

Once you have created the script, you will need to make it executable with:

sudo chmod +x /opt/bin/android-emulator-control.sh

With the script executable, to start the emulator, just do a /opt/bin/android-emulator-control.sh start. To stop it is /opt/bin/android-emulator-control.sh stop.

Installing monit

Next is to install monit if it is not already installed. On a Ubuntu Server, this can be accomplished with:

sudo apt-get install monit

Configuring monit

The next step is to get the monit daemon configured. The general configuration file is located at /etc/monit/monitrc. To edit it, you will need to:

sudo vi /etc/monit/monitrc

You will need to enable or add the following lines in order to make it accessible on the web. Replace the IP address 10.0.0.44 with your server’s IP address, and the 10.0.0.0/24 with the subnet that you want to be able to access the monitor program (e.g. to give everyone in the 192.168.0 subnet access, make it 192.168.0.0/24):

 set httpd port 2812 and
    use address 10.0.0.44
    allow admin:monit
    allow 10.0.0.0/24

Save the file and exit. Next, you will need to create a configuration file for monit that will monitor the Android emulator. To do that, create a new file called /etc/monit/conf.d/android-emulator and add the following code:

check process emulator-5556 with pidfile /tmp/android-emulator-5556.pid
   start program = "/opt/bin/android-emulator-control.sh start"
      as uid bamboo and gid bamboo
   stop program = "/opt/bin/android-emulator-control.sh stop"
      as uid bamboo and gid bamboo
   if failed host localhost port 5556 then restart

Save the file and exit. This configuration file will tell monit to monitor the emulator PID, and restart the program if it finds it is no longer running. It also is configured to run the process as user bamboo with the group bamboo. This is because the bamboo user actually owns the emulator SDK files, and is the only user who is able to run the emulator and other Android commands.

Checking that monit is Running

Okay, now for the moment of truth. If you have configured monit correctly, it is time to start it up and see if it runs the emulator correctly. To start up monit:

sudo /etc/init.d/monit stop
sudo /etc/init.d/monit start

When it is restarted, aim your browser at the port 2812 on your CI server. You should be prompted for a username and password. Use the username and password that you defined in the monitrc file. You should be greeted with a screen that displays the status of your system, and the status of the emulator:

monit-main-screen

If you click on emulator-5556 you will be taken to a screen with more information about the process:

monit-emulator-options

As you can see, there are options here to start and stop the emulator. This is really useful if Bamboo says that there was an error with the emulator. You can use this web screen to monitor its status when you are troubleshooting.

Configuring Bamboo to Test the Project

Okay, the hard part is done. The final step is to configure the Bamboo project to execute the tests. Create a new Script task after code checkout has been completed and call it Run Tests on Emulator. You will need to pass the applicable ANDROID_HOME path. Simply create a script task, and add the following command to it:

./gradlew connectedAndroidTest

That’s all there is to it. Here is a screenshot of my Bamboo task that does this:

bamboo-script-config-android

You will also need a JUnit Parser task in order to pick up the results of the tests. Add this as another task right after the test task. Call it Parse Test Results. You will need to specify the path to the generated XML reports. For Android projects, they are placed in the app/build directory under **/app/build/outputs/androidTest-results/connected/*.xml:

junit-parser-config

Run the Tests!

With everything set up, you should now be able to run the tests. If everything works correctly, the results of your run should be available in Bamboo:

bamboo-successful-android-build

Here you can see my 3 tests ran correctly. These were tests that required instantiating an Activity and checking properties of various objects.

Troubleshooting

Bamboo Throws Errors

The most common error is that when the tests are run, the emulator isn’t up and running. If you look at the gradle output in Bamboo, you will see messages similar to the following:

:app:connectedAndroidTestDebug FAILED
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Execution failed for task ':app:connectedAndroidTestDebug'.
> com.android.builder.testing.api.DeviceException: java.lang.RuntimeException: No connected devices!

The solution here is to make sure the emulator is actually up and running. If the above script is not working for you, you may need to take another approach to running the emulator.

The Emulator Won’t Run

There are many reasons why the emulator will not run from the simple start/stop script outlined above. For most of them, setting the path to the SDK is key, and making sure the AVD images are where the emulator command expects to see them. Your best bet is to look at the log file that is generated when the start/stop script is called. In my example above, it will be at /tmp/android-emulator-5556.log. It will usually give you enough error information to help you figure out what went wrong. Also, be sure that you have created the emulator image with the android command with the user that has access to your Android SDK.

Multiple Build Agents

While the above works well if you have a single build agent, things get more complicated if you have multiple build agents running on the same machine. If more than one build agent tries to access the emulator at a time, you will probably get some unexpected results (crashes or failures). My suggestion would be to have several different remote agents. On each remote agent machine, you can have an emulator running using the process I described above.

Alternatively, define only a single agent with the capability to run Android tests. This will mean that on a busy server you will experience a backlog of jobs that need to be built, but at least you will not cause collisions with the emulator and different build agents. As a compromise, creating a single nightly run that executes these type of emulator tests on your development branch may be a good alternative to running them every time a branch is created or updated.

Wrapping Up

In this post, I discussed a solution for running Android tests on a Bamboo CI server. It involves running an emulator in the background so that the build agent can connect to it when it needs to run tests. Other CI servers have interesting plugins that help with the automated configuration and provisioning of emulators on the fly (for example, Jenkins has an Android Emulator Plugin that helps out with this task). The solution I describe above used a simple command called monit to run an emulator in the background.

Hopefully you have been able to implement the steps above, and have Bamboo building your latest Android project! Feel free to asks questions or leave comments in the comment section below.

Play Framework Custom Integration Test Configuration

I’ve been using the Play Framework 2.3 a lot recently for Java development, and have been finding it really easy to use. However, sometimes documentation is scarce, or doesn’t have an example of what it is I am looking for. In this blog post, I discuss how to create a custom testing configuration, and specify a custom path for your test source files.

The Problem: Integration Tests Too Slow

I was working on a web framework and got to the point where integration tests were taking more than a few seconds to run. This was expected, since I was performing end-to-end testing of several complex components. This isn’t a problem by itself, however, when performing test driven development I like to have the test runner executing in the background, picking up new tests and changes to existing tests as I write. For those of you who aren’t familiar with it, you can have the Typesafe Activator do this with the following command:

./activator ~test

The problem was that the default test runner was executing all of the tests, which was a pain since I would have to wait for all of the slow tests to run to see if there was a problem with any changes I made. What I needed was a way to separate out the unit tests from the more complicated integration tests.

The Start of a Solution

As it turns out, since the Activator uses sbt under the covers, it has access to a default integration testing configuration called IntegrationTest. All you need to do is change your root definition to include a configuration for IntegrationTest in build.sbt, and invoke the default integration test settings:

lazy val root = (project in file(".")).enablePlugins(PlayJava) configs(IntegrationTest)
 
Defaults.itSettings

This defines a new testing configuration called it. To run the integration tests all you would need to do is invoke:

it:test

While this put me on the right track, the problem is that the default IntegrationTest configuration defines the location of integration tests to be /src/it (you can see more about where these directories are by running show it:sourceDirectory and inspect it:sourceDirectory from an Activator prompt). Unfortunately, this layout doesn’t fit well with the rest of the Play Framework source tree. On top of that, when I moved my integration tests into the new directory layout, actually running the integration tests resulted in many hundreds of symbol errors such as:

[error]                 assertThat(contentAsString(result)).isEqualTo("mystring");
[error]                            ^
[error]   symbol: method contentAsString(Result)
[error] /projects/myproject/it/controllers/ControllerTest.java:120: error: cannot find symbol
[error]         running(fakeApplication(inMemoryDatabase()), new Runnable() {
[error]                                 ^
[error]   symbol:   method inMemoryDatabase()
[error] /projects/myproject/it/controllers/ControllerTest.java:98: error: cannot find symbol
[error]                         new FakeRequest(Helpers.POST, "/controller/create")
[error]                             ^
[error]   symbol: class FakeRequest
[error] /projects/myproject/it/controllers/ControllerTest.java:202: error: cannot find symbol

The problem is that the IntegrationTest configuration wasn’t pulling in the dependencies that were defined from the default Test configuration. That was a problem.

The Full Solution

Rather than play around with the IntegrationTest configuration, I figured it would be better to define my own custom configuration. The first step was to define my own configuration called ITest in the build.sbt file. I wanted this to be defined based off of the Test configuration, so I made my configuration extend it:

lazy val root = (project in file(".")).enablePlugins(PlayJava) configs(ITest) settings( inConfig(ITest)(Defaults.testSettings) : _*)
 
lazy val ITest = config("it") extend(Test)

So far so good. This created my new integration testing configuration called ITest. The config("it") portion of the second line defines how you can access that configuration from the Activator. I wanted to keep it simple, and just run it:test for my integration tests (like in the section above), so I kept it as "it".

The next hurdle was to define the proper directory layout. With a Play Framework 2.3.x application, the (partial) directory layout is as follows:

  • app is where all of your application source code (controllers, views, models, etc) goes.
  • test is where all of your testing source code goes.

What I wanted to do was create a new directory called it where all of the source code for integration tests would live. To do that, I added the following line to the build.sbt file:

sourceDirectory in ITest := baseDirectory.value / "/it"

This sort of worked. The problem was that it created other directories under /it for Java and Scala sources. Not quite what I wanted. So I defined three other source directories as follows:

javaSource in ITest := baseDirectory.value / "/it"
 
resourceDirectory in ITest := baseDirectory.value / "/it/resources"
 
scalaSource in ITest := baseDirectory.value / "/it"

This basically mashes both the Scala and Java sources into the /it directory, and names a specific directory for resources under /it/resources.

After syncing the changes to build.sbt, IntelliJ built the new directory structure for me. I moved my integration tests into the /it directory and ran the integration tests only:

[MyProject] $ it:test
[info] Updating {file:/projects/myproject/}root...
[info] Resolving jline#jline;2.11 ...
[info] Done updating.
[info] Passed: Total 57, Failed 0, Errors 0, Passed 57
[success] Total time: 49 s, completed 8-May-2015 1:40:58 AM

Just to be sure it would run the old unit tests (and only the unit tests – I knew there should only be 229 unit tests):

[MyProject] $ test
[info] Passed: Total 229, Failed 0, Errors 0, Passed 229
[success] Total time: 18 s, completed 8-May-2015 1:42:35 AM

Perfect.

Wrapping Up

To create a custom integration testing configuration, I only had to change a few lines in the build.sbt file. All together:

lazy val root = (project in file(".")).enablePlugins(PlayJava) configs(ITest) settings( inConfig(ITest)(Defaults.testSettings) : _*)
 
lazy val ITest = config("it") extend(Test)
 
sourceDirectory in ITest := baseDirectory.value / "/it"
 
javaSource in ITest := baseDirectory.value / "/it"
 
resourceDirectory in ITest := baseDirectory.value / "/it/resources"
 
scalaSource in ITest := baseDirectory.value / "/it"

Now, running integration tests is as easy as it:test while running unit tests remains as test.