Android and Bamboo – Testing Using the Android Emulator on a CI Server

In a previous post, I discussed how to set up Atlassian Bamboo to build unit tests as part of an Android project. In that post, I went through the steps needed to install the Android SDK on a Bamboo build server, and configure a Bamboo build project to build and test the codebase.

One of the assumptions that I made in the previous post was that all of the unit tests were written with a package called Robolectric (see my post on Android Unit Testing with Robolectric for details on how to set up and use it). Why use Robolectric? Well, it allows you to test your project without having to run the tests on an Android emulator. As you may already know, the emulator – while good – can be slow. However, there are many times when you need to test on an actual Android device. In this post, I will discuss how to run an emulator on your CI server, and configure Bamboo to run Android tests directly in the emulated device.

Assumptions

In this post, I’m going to make several assumptions:

  1. Your Bamboo server is running a Linux OS.
  2. You have already installed the SDK on the Bamboo server, and that it has the components you need to build and support your Android application. If you don’t already have this set up, see my previous post on how to do this.
  3. You are familiar enough with Bamboo that you know how to configure a project, and add several tasks to it.
  4. You have access to a shell on the build server so that you can define a few things that Bamboo needs.
  5. You want to run tests that actually require an emulator (or device) to run. Tests such as ActivityInstrumentationTestCase2 and ProviderTestCase2 are good examples of tests that need to be run on an emulator or device.

The Solution

Implementing this solution involves the following steps:

  1. Define an emulated device.
  2. Start a headless emulator in the background, and monitor it.
  3. In Bamboo, add a task to run the tests on the emulator.

Define an Emulated Device

You need to shell into your build server now. Once there, you need to define an emulator with the configuration your application requires. In my case, I needed an emulator that supported API level 19 (KitKat). In the example below, I’m also using the x86 image for the emulator. I called my new emulator android_x86. Make sure you switch to the user that has access to the Android SDK that you installed.

android create avd --force -n android_x86 -t android-19 --abi x86

Side-note: why use the x86 image? If you are have either an Intel or AMD processor on Linux, the x86 image can be accelerated as long as you have KVM installed. See my post on Android Emulator VM Acceleration on AMD for details on how to install KVM.

Starting and Monitoring a Headless Emulator

For this task, we will create a start/stop script for an Android emulator, and install and use the Linux program monit to spin up the emulator and make sure it continues to run. The monit program is a great monitoring program that will be able to see if an emulator is running, and if not, start one up in headless mode. Plus, it has a web interface that will let us start, stop, and monitor the status of the running Android emulator from within a browser.

An Emulator Start/Stop Script

The first thing to do is to create a script that will be responsible for starting and stopping the emulator in headless mode. What is headless mode? Usually a build server does not have a monitor attached to it, and thus anything that tries to open a window on the display will error out. Operating the emulator in headless mode will prevent that from happening, since it won’t try and display an emulator window. A few blogs out on the net have good instructions on how to do this (for example, Paul Estrada’s post on defining a headless emulator for Travis CI).

The script below acts as a simple start/stop script. I named the script android-emulator-control.sh and put it in /opt/bin for easy access. You will need to edit ANDROID_HOME, ANDROID_SDK_HOME, and the EMULATOR_PORT depending on where your Android environment is configured, and what port you want to run it on. You’ll also need to edit the AVD_NAME to match the name of the AVD you created in the previous step. The script will start up an emulator on the designated port (in this case port 5556). It will also create a process identification file at /tmp/android-emulator-5556.pid and a log file at /tmp/android-emulator-5556.log.

#!/bin/bash
export ANDROID_HOME=/opt/android/android-sdk-linux
export ANDROID_SDK_HOME=/opt/android/android-sdk-linux
EMULATOR_PORT=5556
PID_FILE="/tmp/android-emulator-$EMULATOR_PORT.pid"
LOG_FILE="/tmp/android-emulator-$EMULATOR_PORT.log"
EXEC_FILE="$ANDROID_HOME/tools/emulator"
AVD_NAME="android_x86"
 
case $1 in
    start)
        $EXEC_FILE -avd $AVD_NAME -port $EMULATOR_PORT -no-skin -no-audio -no-window \
            -qemu -m 512 -enable-kvm > $LOG_FILE 2>&1 & echo $! > $PID_FILE
        ;;
    stop)
        kill `cat $PID_FILE`
        ;;
    *)
        echo "usage: android-emulator-control.sh {start|stop}" ;;
esac
exit 0

Once you have created the script, you will need to make it executable with:

sudo chmod +x /opt/bin/android-emulator-control.sh

With the script executable, to start the emulator, just do a /opt/bin/android-emulator-control.sh start. To stop it is /opt/bin/android-emulator-control.sh stop.

Installing monit

Next is to install monit if it is not already installed. On a Ubuntu Server, this can be accomplished with:

sudo apt-get install monit

Configuring monit

The next step is to get the monit daemon configured. The general configuration file is located at /etc/monit/monitrc. To edit it, you will need to:

sudo vi /etc/monit/monitrc

You will need to enable or add the following lines in order to make it accessible on the web. Replace the IP address 10.0.0.44 with your server’s IP address, and the 10.0.0.0/24 with the subnet that you want to be able to access the monitor program (e.g. to give everyone in the 192.168.0 subnet access, make it 192.168.0.0/24):

 set httpd port 2812 and
    use address 10.0.0.44
    allow admin:monit
    allow 10.0.0.0/24

Save the file and exit. Next, you will need to create a configuration file for monit that will monitor the Android emulator. To do that, create a new file called /etc/monit/conf.d/android-emulator and add the following code:

check process emulator-5556 with pidfile /tmp/android-emulator-5556.pid
   start program = "/opt/bin/android-emulator-control.sh start"
      as uid bamboo and gid bamboo
   stop program = "/opt/bin/android-emulator-control.sh stop"
      as uid bamboo and gid bamboo
   if failed host localhost port 5556 then restart

Save the file and exit. This configuration file will tell monit to monitor the emulator PID, and restart the program if it finds it is no longer running. It also is configured to run the process as user bamboo with the group bamboo. This is because the bamboo user actually owns the emulator SDK files, and is the only user who is able to run the emulator and other Android commands.

Checking that monit is Running

Okay, now for the moment of truth. If you have configured monit correctly, it is time to start it up and see if it runs the emulator correctly. To start up monit:

sudo /etc/init.d/monit stop
sudo /etc/init.d/monit start

When it is restarted, aim your browser at the port 2812 on your CI server. You should be prompted for a username and password. Use the username and password that you defined in the monitrc file. You should be greeted with a screen that displays the status of your system, and the status of the emulator:

monit-main-screen

If you click on emulator-5556 you will be taken to a screen with more information about the process:

monit-emulator-options

As you can see, there are options here to start and stop the emulator. This is really useful if Bamboo says that there was an error with the emulator. You can use this web screen to monitor its status when you are troubleshooting.

Configuring Bamboo to Test the Project

Okay, the hard part is done. The final step is to configure the Bamboo project to execute the tests. Create a new Script task after code checkout has been completed and call it Run Tests on Emulator. You will need to pass the applicable ANDROID_HOME path. Simply create a script task, and add the following command to it:

./gradlew connectedAndroidTest

That’s all there is to it. Here is a screenshot of my Bamboo task that does this:

bamboo-script-config-android

You will also need a JUnit Parser task in order to pick up the results of the tests. Add this as another task right after the test task. Call it Parse Test Results. You will need to specify the path to the generated XML reports. For Android projects, they are placed in the app/build directory under **/app/build/outputs/androidTest-results/connected/*.xml:

junit-parser-config

Run the Tests!

With everything set up, you should now be able to run the tests. If everything works correctly, the results of your run should be available in Bamboo:

bamboo-successful-android-build

Here you can see my 3 tests ran correctly. These were tests that required instantiating an Activity and checking properties of various objects.

Troubleshooting

Bamboo Throws Errors

The most common error is that when the tests are run, the emulator isn’t up and running. If you look at the gradle output in Bamboo, you will see messages similar to the following:

:app:connectedAndroidTestDebug FAILED
 
FAILURE: Build failed with an exception.
 
* What went wrong:
Execution failed for task ':app:connectedAndroidTestDebug'.
> com.android.builder.testing.api.DeviceException: java.lang.RuntimeException: No connected devices!

The solution here is to make sure the emulator is actually up and running. If the above script is not working for you, you may need to take another approach to running the emulator.

The Emulator Won’t Run

There are many reasons why the emulator will not run from the simple start/stop script outlined above. For most of them, setting the path to the SDK is key, and making sure the AVD images are where the emulator command expects to see them. Your best bet is to look at the log file that is generated when the start/stop script is called. In my example above, it will be at /tmp/android-emulator-5556.log. It will usually give you enough error information to help you figure out what went wrong. Also, be sure that you have created the emulator image with the android command with the user that has access to your Android SDK.

Multiple Build Agents

While the above works well if you have a single build agent, things get more complicated if you have multiple build agents running on the same machine. If more than one build agent tries to access the emulator at a time, you will probably get some unexpected results (crashes or failures). My suggestion would be to have several different remote agents. On each remote agent machine, you can have an emulator running using the process I described above.

Alternatively, define only a single agent with the capability to run Android tests. This will mean that on a busy server you will experience a backlog of jobs that need to be built, but at least you will not cause collisions with the emulator and different build agents. As a compromise, creating a single nightly run that executes these type of emulator tests on your development branch may be a good alternative to running them every time a branch is created or updated.

Wrapping Up

In this post, I discussed a solution for running Android tests on a Bamboo CI server. It involves running an emulator in the background so that the build agent can connect to it when it needs to run tests. Other CI servers have interesting plugins that help with the automated configuration and provisioning of emulators on the fly (for example, Jenkins has an Android Emulator Plugin that helps out with this task). The solution I describe above used a simple command called monit to run an emulator in the background.

Hopefully you have been able to implement the steps above, and have Bamboo building your latest Android project! Feel free to asks questions or leave comments in the comment section below.

Gradle and SonarQube

If you are like me and like developing using a Test Driven Development (TDD) approach, then you need the ability to examine your code and test coverage. In a past article, I discussed how to use SonarQube to perform Python code inspection allowing you to see code test coverage. In this article, I’ll look at how to set up Gradle to talk to SonarQube.

The SonarRunner

SonarQube has a companion tool called the SonarRunner. The runner is what actually runs the unit tests (or integration tests) on your software, and then reports it to the SonarQube database. One of the nice things about the SonarRunner is that you can add it to just about any project that can be analyzed by SonarQube – regardless of whatever other lifecycle or build tools you may use. All you need to do is add a sonar-runner.properties file with the required information.

One problem however, is that the SonarRunner is yet another tool that is needed during the build process. If you maintain a continuous integration environment where, say, your nightly builds include an inspection run, then you know the pain of having to maintain yet another standalone binary package. Luckily, Gradle has a SonarRunner plugin that is incubating, and is stable enough to use.

Setting Up the Gradle Plugin

The SonarRunner plugin has been included in most recent distributions of Gradle. Applying the plugin requires one line:

apply plugin: sonar-runner

In order to get the runner talking to your local installation of Sonar, you will also need a sonarRunner configuration section in your build.gradle file (note – if you have an older version of Gradle this may not work for you, since older versions used slightly different properties). A minimal set of configuration properties is as follows:

sonarRunner {
    sonarProperties {
        property "sonar.host.url", "http://10.0.0.60:9000"
        property "sonar.jdbc.url", "jdbc:postgresql://10.0.0.60:5432/sonar"
        property "sonar.jdbc.driverClassName", "org.postgresql.Driver"
        property "sonar.jdbc.username", "mySonarUsername"
        property "sonar.jdbc.password", "mySonarPassword"
        property "sonar.projectKey", "MyJavaProject"
        property "sonar.projectName", "My Java Project"
        property "sonar.projectVersion", android.defaultConfig.versionName
        property "sonar.language", "java"
        property "sonar.sources", "src/main"
        property "sonar.binaries", "build"
    }
}

Here are some of the settings in more detail:

  • sonar.host.url – this is the URL to your Sonar host. Note that if you are trying to use a host that is protected by SSL, you will need some additional configuration information.
  • sonar.jdbc.url – this is the JDBC (database) URL. In my case, I was using PostgreSQL server, so I used the postgresql JDBC syntax. The 10.0.0.60 is my internal IP address, and 5432 is the port that the database listens on. Note that in the case of PostgreSQL, you need to explicitly tell it the addresses to listen on in your postgres.conf file, otherwise it defaults to listening on port 5432 only for localhost.
  • sonar.jdbc.driverClassName – should be fairly obvious that this is the driver it should be using for the database connection.
  • sonar.jdbc.username – this is the username used to access the database – not a username that you set up to access the front-end application (if you have authentication for SonarQube turned on).
  • sonar.jdbc.password – this is the password used to access the database – again, not a front-end application username password.
  • sonar.projectKey – a unique name used to identify the project.
  • sonar.projectName – the name of the project, more readable for us humans.
  • sonar.projectVersion – should be fairly obvious. In this instance, I have a second set of `android` properties which are describing my project. Instead of duplicating the version number, I just used it here instead.
  • sonar.language – tells Sonar what language you are analyzing.
  • sonar.sources – where the source files for the project are located. Make sure you keep this path separate from your test cases, otherwise you will have your tests included in the analysis.
  • sonar.binaries – where build artifacts are located.

Configuring JaCoCo

While the above configuration will get you up and running, you will be missing coverage information from your unit tests. To generate coverage reports, you will also need to apply the JaCoCo plugin:

apply plugin: jacoco

Then, you need to set two more Sonar Runner properties as follows (add them below sonar.binaries):

property 'sonar.jacoco.reportPath', "${buildDir}/jacoco/testDebug.exec"
property 'sonar.junit.reportsPath', "${buildDir}/test-results"

You may have to adjust these depending on where jacoco is actually putting your results. Look for the exec file and the test-results paths after you run a build.

All Together

Putting it all together:

apply plugin: sonar-runner
apply plugin: jacoco
 
sonarRunner {
    sonarProperties {
        property "sonar.host.url", "http://10.0.0.60:9000"
        property "sonar.jdbc.url", "jdbc:postgresql://10.0.0.60:5432/sonar"
        property "sonar.jdbc.driverClassName", "org.postgresql.Driver"
        property "sonar.jdbc.username", "mySonarUsername"
        property "sonar.jdbc.password", "mySonarPassword"
        property "sonar.projectKey", "MyJavaProject"
        property "sonar.projectName", "My Java Project"
        property "sonar.projectVersion", android.defaultConfig.versionName
        property "sonar.language", "java"
        property "sonar.sources", "src/main"
        property "sonar.binaries", "build"
        property 'sonar.jacoco.reportPath', "${buildDir}/jacoco/testDebug.exec"
        property 'sonar.junit.reportsPath', "${buildDir}/test-results"
    }
}

You can now check out whether or not Gradle is configured correctly by looking at the tasks:

./gradlew tasks

You should see a new task called sonarRunner:

sonarRunner - Analyzes project ':app' and its subprojects with Sonar Runner.

To generate reports, you do the following:

./gradlew clean test sonarRunner

Gradle will clean the project, rebuild and run the unit tests. It will then perform a Sonar analysis, and post the results to your SonarQube instance. If everything is successful, you should see output similar to the following from Gradle:

17:30:40.937 INFO  - Sensor JaCoCoSensor...
17:30:40.965 INFO  - Analysing /home/thomas/AndroidStudioProjects/MyJavaProject/app/build/jacoco/testDebug.exec
17:30:41.598 INFO  - No information about coverage per test.
17:30:41.598 INFO  - Sensor JaCoCoSensor done: 661 ms
17:30:42.955 INFO  - Execute decorators...
17:30:45.010 INFO  - Store results in database
17:30:45.133 INFO  - ANALYSIS SUCCESSFUL, you can browse http://10.0.0.60:9000/dashboard/index/MyJavaProject
17:30:45.205 INFO  - Executing post-job class org.sonar.plugins.core.issue.notification.SendIssueNotificationsPostJob
17:30:45.210 INFO  - Executing post-job class org.sonar.plugins.core.batch.IndexProjectPostJob
17:30:45.266 INFO  - Executing post-job class org.sonar.plugins.dbcleaner.ProjectPurgePostJob
17:30:45.289 INFO  - -> Keep one snapshot per day between 2014-09-19 and 2014-10-16
17:30:45.291 INFO  - -> Keep one snapshot per week between 2013-10-18 and 2014-09-19
17:30:45.292 INFO  - -> Keep one snapshot per month between 2009-10-23 and 2013-10-18
17:30:45.294 INFO  - -> Delete data prior to: 2009-10-23
17:30:45.302 INFO  - -> Clean MyJavaProject [id=160]
17:30:45.309 INFO  - Clean snapshot 5368
 
BUILD SUCCESSFUL

And of course, browsing to your Sonar instance should reveal some good information about your project.

Summary

In this post, I discussed how to configure Gradle to run a Sonar analysis on your code, and post the results back to your SonarQube instance. In a future post, I’ll look at more of the SonarQube analysis results, and talk about common fixes for various problems.