Android and Bamboo – Testing Using the Android Emulator on a CI Server

In a previous post, I discussed how to set up Atlassian Bamboo to build unit tests as part of an Android project. In that post, I went through the steps needed to install the Android SDK on a Bamboo build server, and configure a Bamboo build project to build and test the codebase.

One of the assumptions that I made in the previous post was that all of the unit tests were written with a package called Robolectric (see my post on Android Unit Testing with Robolectric for details on how to set up and use it). Why use Robolectric? Well, it allows you to test your project without having to run the tests on an Android emulator. As you may already know, the emulator – while good – can be slow. However, there are many times when you need to test on an actual Android device. In this post, I will discuss how to run an emulator on your CI server, and configure Bamboo to run Android tests directly in the emulated device.

Assumptions

In this post, I’m going to make several assumptions:

  1. Your Bamboo server is running a Linux OS.
  2. You have already installed the SDK on the Bamboo server, and that it has the components you need to build and support your Android application. If you don’t already have this set up, see my previous post on how to do this.
  3. You are familiar enough with Bamboo that you know how to configure a project, and add several tasks to it.
  4. You have access to a shell on the build server so that you can define a few things that Bamboo needs.
  5. You want to run tests that actually require an emulator (or device) to run. Tests such as ActivityInstrumentationTestCase2 and ProviderTestCase2 are good examples of tests that need to be run on an emulator or device.

The Solution

Implementing this solution involves the following steps:

  1. Define an emulated device.
  2. Start a headless emulator in the background, and monitor it.
  3. In Bamboo, add a task to run the tests on the emulator.

Define an Emulated Device

You need to shell into your build server now. Once there, you need to define an emulator with the configuration your application requires. In my case, I needed an emulator that supported API level 19 (KitKat). In the example below, I’m also using the x86 image for the emulator. I called my new emulator android_x86. Make sure you switch to the user that has access to the Android SDK that you installed.

android create avd --force -n android_x86 -t android-19 --abi x86

Side-note: why use the x86 image? If you are have either an Intel or AMD processor on Linux, the x86 image can be accelerated as long as you have KVM installed. See my post on Android Emulator VM Acceleration on AMD for details on how to install KVM.

Starting and Monitoring a Headless Emulator

For this task, we will create a start/stop script for an Android emulator, and install and use the Linux program monit to spin up the emulator and make sure it continues to run. The monit program is a great monitoring program that will be able to see if an emulator is running, and if not, start one up in headless mode. Plus, it has a web interface that will let us start, stop, and monitor the status of the running Android emulator from within a browser.

An Emulator Start/Stop Script

The first thing to do is to create a script that will be responsible for starting and stopping the emulator in headless mode. What is headless mode? Usually a build server does not have a monitor attached to it, and thus anything that tries to open a window on the display will error out. Operating the emulator in headless mode will prevent that from happening, since it won’t try and display an emulator window. A few blogs out on the net have good instructions on how to do this (for example, Paul Estrada’s post on defining a headless emulator for Travis CI).

The script below acts as a simple start/stop script. I named the script android-emulator-control.sh and put it in /opt/bin for easy access. You will need to edit ANDROID_HOME, ANDROID_SDK_HOME, and the EMULATOR_PORT depending on where your Android environment is configured, and what port you want to run it on. You’ll also need to edit the AVD_NAME to match the name of the AVD you created in the previous step. The script will start up an emulator on the designated port (in this case port 5556). It will also create a process identification file at /tmp/android-emulator-5556.pid and a log file at /tmp/android-emulator-5556.log.

#!/bin/bash
export ANDROID_HOME=/opt/android/android-sdk-linux
export ANDROID_SDK_HOME=/opt/android/android-sdk-linux
EMULATOR_PORT=5556
PID_FILE="/tmp/android-emulator-$EMULATOR_PORT.pid"
LOG_FILE="/tmp/android-emulator-$EMULATOR_PORT.log"
EXEC_FILE="$ANDROID_HOME/tools/emulator"
AVD_NAME="android_x86"
 
case $1 in
    start)
        $EXEC_FILE -avd $AVD_NAME -port $EMULATOR_PORT -no-skin -no-audio -no-window \
            -qemu -m 512 -enable-kvm > $LOG_FILE 2>&1 & echo $! > $PID_FILE
        ;;
    stop)
        kill `cat $PID_FILE`
        ;;
    *)
        echo "usage: android-emulator-control.sh {start|stop}" ;;
esac
exit 0

Once you have created the script, you will need to make it executable with:

sudo chmod +x /opt/bin/android-emulator-control.sh

With the script executable, to start the emulator, just do a /opt/bin/android-emulator-control.sh start. To stop it is /opt/bin/android-emulator-control.sh stop.

Installing monit

Next is to install monit if it is not already installed. On a Ubuntu Server, this can be accomplished with:

sudo apt-get install monit

Configuring monit

The next step is to get the monit daemon configured. The general configuration file is located at /etc/monit/monitrc. To edit it, you will need to:

sudo vi /etc/monit/monitrc

You will need to enable or add the following lines in order to make it accessible on the web. Replace the IP address 10.0.0.44 with your server’s IP address, and the 10.0.0.0/24 with the subnet that you want to be able to access the monitor program (e.g. to give everyone in the 192.168.0 subnet access, make it 192.168.0.0/24):

 set httpd port 2812 and
    use address 10.0.0.44
    allow admin:monit
    allow 10.0.0.0/24

Save the file and exit. Next, you will need to create a configuration file for monit that will monitor the Android emulator. To do that, create a new file called /etc/monit/conf.d/android-emulator and add the following code:

check process emulator-5556 with pidfile /tmp/android-emulator-5556.pid
   start program = "/opt/bin/android-emulator-control.sh start"
      as uid bamboo and gid bamboo
   stop program = "/opt/bin/android-emulator-control.sh stop"
      as uid bamboo and gid bamboo
   if failed host localhost port 5556 then restart

Save the file and exit. This configuration file will tell monit to monitor the emulator PID, and restart the program if it finds it is no longer running. It also is configured to run the process as user bamboo with the group bamboo. This is because the bamboo user actually owns the emulator SDK files, and is the only user who is able to run the emulator and other Android commands.

Checking that monit is Running

Okay, now for the moment of truth. If you have configured monit correctly, it is time to start it up and see if it runs the emulator correctly. To start up monit:

sudo /etc/init.d/monit stop
sudo /etc/init.d/monit start

When it is restarted, aim your browser at the port 2812 on your CI server. You should be prompted for a username and password. Use the username and password that you defined in the monitrc file. You should be greeted with a screen that displays the status of your system, and the status of the emulator:

monit-main-screen

If you click on emulator-5556 you will be taken to a screen with more information about the process:

monit-emulator-options

As you can see, there are options here to start and stop the emulator. This is really useful if Bamboo says that there was an error with the emulator. You can use this web screen to monitor its status when you are troubleshooting.

Configuring Bamboo to Test the Project

Okay, the hard part is done. The final step is to configure the Bamboo project to execute the tests. Create a new Script task after code checkout has been completed and call it Run Tests on Emulator. You will need to pass the applicable ANDROID_HOME path. Simply create a script task, and add the following command to it:

./gradlew connectedAndroidTest

That’s all there is to it. Here is a screenshot of my Bamboo task that does this:

bamboo-script-config-android

You will also need a JUnit Parser task in order to pick up the results of the tests. Add this as another task right after the test task. Call it Parse Test Results. You will need to specify the path to the generated XML reports. For Android projects, they are placed in the app/build directory under **/app/build/outputs/androidTest-results/connected/*.xml:

junit-parser-config

Run the Tests!

With everything set up, you should now be able to run the tests. If everything works correctly, the results of your run should be available in Bamboo:

bamboo-successful-android-build

Here you can see my 3 tests ran correctly. These were tests that required instantiating an Activity and checking properties of various objects.

Troubleshooting

Bamboo Throws Errors

The most common error is that when the tests are run, the emulator isn’t up and running. If you look at the gradle output in Bamboo, you will see messages similar to the following:

:app:connectedAndroidTestDebug FAILED
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Execution failed for task ':app:connectedAndroidTestDebug'.
> com.android.builder.testing.api.DeviceException: java.lang.RuntimeException: No connected devices!

The solution here is to make sure the emulator is actually up and running. If the above script is not working for you, you may need to take another approach to running the emulator.

The Emulator Won’t Run

There are many reasons why the emulator will not run from the simple start/stop script outlined above. For most of them, setting the path to the SDK is key, and making sure the AVD images are where the emulator command expects to see them. Your best bet is to look at the log file that is generated when the start/stop script is called. In my example above, it will be at /tmp/android-emulator-5556.log. It will usually give you enough error information to help you figure out what went wrong. Also, be sure that you have created the emulator image with the android command with the user that has access to your Android SDK.

Multiple Build Agents

While the above works well if you have a single build agent, things get more complicated if you have multiple build agents running on the same machine. If more than one build agent tries to access the emulator at a time, you will probably get some unexpected results (crashes or failures). My suggestion would be to have several different remote agents. On each remote agent machine, you can have an emulator running using the process I described above.

Alternatively, define only a single agent with the capability to run Android tests. This will mean that on a busy server you will experience a backlog of jobs that need to be built, but at least you will not cause collisions with the emulator and different build agents. As a compromise, creating a single nightly run that executes these type of emulator tests on your development branch may be a good alternative to running them every time a branch is created or updated.

Wrapping Up

In this post, I discussed a solution for running Android tests on a Bamboo CI server. It involves running an emulator in the background so that the build agent can connect to it when it needs to run tests. Other CI servers have interesting plugins that help with the automated configuration and provisioning of emulators on the fly (for example, Jenkins has an Android Emulator Plugin that helps out with this task). The solution I describe above used a simple command called monit to run an emulator in the background.

Hopefully you have been able to implement the steps above, and have Bamboo building your latest Android project! Feel free to asks questions or leave comments in the comment section below.

Writing a Play Framework Plugin in Java

I really like using the Play Framework to write simple web applications and REST interfaces. One of the interesting things about the Play Framework is that it is meant to be both stateless and asynchronous. The stateless nature of the framework makes it easy to scale very quickly, since all you need to do is deploy multiple instances of your web application in order to serve larger loads.

Sometimes however, you need to introduce some sort of mechanism to create a global variable or object. For example, if your application needs to make an expensive database connection, sometimes it is better to create a single object (or pool of objects) that can handle your database connection, and have your controllers pick those up instead of generating new ones each time a REST endpoint or webpage is served. In this post, I’ll look at creating a Play Framework plugin, and how you can maintain a global state with that plugin.

The Problem

There are times that I have an Android emulator running and I want to issue it external events programmatically. The trick here is that I want to do it from within the emulator. More specifically, when I’m running tests for an application that receives SMS messages, it’s nice to be able to trigger an inbound SMS on demand. You can trigger SMS messages by opening a telnet session to your emulator and issuing the sms command. For example, assuming your emulator is running on port 5554, a typical session might look like this:

# telnet localhost 5554
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Android Console: type 'help' for a list of commands
OK
sms send 5550100 This is a test message!!
OK

The running emulator instantly receives it:

sms-message-android

If I’m doing a lot of automated testing directly on the Android emulator, I want the emulated Android test cases to easily trigger an incoming SMS by itself. To get this functionality, I want to spin up a Play application on my testing machine that can open telnet sessions to the running emulators. The application would accept REST calls to generate the required incoming SMS, and post them to the correct virtual device.

Sidenote: how do the tests running on the Android emulator know what emulator they are running on? Easy – the last 4 digits of the emulated device’s phone number are the telnet port that it is running on. The test case can use TelephonyManager getLine1Number() to read the telephone number, and parse the port number. It then passes the port number on to the Play application when it makes the call to the REST API.

The Solution

To do this, I built a telnet manager class that would connect to the emulator, and then allow me to input commands and capture the output. To connect this telnet manager to my Play application, I needed to keep the telnet manager object around in some sort of global variable and make it accessible, since I didn’t want to connect and disconnect from the running emulator every single time I wanted to run a command.

There are two different approaches that would work for creating this type of global object. The first is to create an Actor that would work through Akka. I didn’t investigate how this was done, but instead opted to create a Play plugin. The benefit of creating a Play plugin is that they are very easy to create. The breakdown for creating the Play plugin is as follows:

  1. Create the class that you want to manage your service.
  2. Create a class that implements play.api.Plugin interface.
  3. Add the plugin to your Play configuration.

Creating the TelnetManager

There isn’t too much to say about this step of the solution. In my case, I have a simple TelnetManager class that I am using to connect to a running emulator. With my class, I can open up a connection to a running emulator, and then issue commands to it. The idea is that the Play framework application can open a connection to an emulator, and keep that connection open between different REST calls.

Since I know it’s bad if two different processes try and use the object at the same time, I used a simple lock variable to block until the resource becomes free. This ensures that two processes don’t try and talk to the same device at the same time.

Creating the Play Framework Plugin

All you need to do is implement the play.api.Plugin interface. Minimally, it looks like this:

package plugins;
 
import models.TelnetManager;
import play.Application;
import play.api.Plugin;
 
public class AndroidSMSMessagerPlugin implements Plugin {
 
    private final TelnetManager sTelnetManager = new TelnetManager();
 
    public AndroidSMSMessagerPlugin(Application app) {
    }
 
    public void onStart() {
        sTelnetManager.start();
    }
 
    public void onStop() {
        sTelnetManager.stop();
    }
 
    public boolean enabled() {
        return true;
    }
 
    public TelnetManager getTelnetManager() {
        return sTelnetManager;
    }
}

From the above code, you can see that there are only 4 methods that you need to implement in your Plugin:

  • onStart – runs any code that you need to run when the plugin is started by the Play framework. In my case, I have a start() function in my TelnetManager class that initializes a connection pool.
  • onStop – runs any code that you need to run when the plugin is stopped. In my case, I have a stop() function that destroys any open connections to any of the emulators, and releases any handles that it held open.
  • enabled – tells the Play framework if the plugin is actually enabled.
  • getTelnetManager – this is where the magic is. This function returns a handle to the globally opened TelnetManager object. With this function, I can use this in any of my controllers to actually get a reference to the global object (more on this below).

With those methods implemented, the last step is to configure your Play application to recognize the plugin.

Adding the Plugin to your Play Configuration

The easiest way to add your new plugin is to create a file in the conf of your project called play.plugins. In it, all you need is the following line:

2000:plugins.AndroidSMSMessagerPlugin

The 2000 acts as a priority level. The lower the number, the sooner the plugin is loaded (values can range from 0 – 10000). This can help you control when your plugin is loaded by the framework.

Accessing the Plugin

With everything plugged into the framework, the last thing to do is to actually access the plugin from a controller. You can do it with the following:

TelnetManager tm = Play.application().plugin(AndroidSMSMessagerPlugin.class).getTelnetManager();

As you can see, I now have a reference to the global TelnetManager object!

Wrapping Up

In this post, I went over how to create a Play framework plugin, and why you might need one. By implementing only 4 methods, you can easily create your own plugin to the Play framework.