On-Device Open Source CI and CD for Mobile Reinvented: Meet Concourse

Concourse is a simple-yet-flexible open source CI system which is maintained and developed by a team of engineers at Pivotal. For the Pivotal Cloud Foundry team in Toronto, which focuses on mobile services, Concourse was a useful upgrade from GoCD, our previous CI/CD system. The flexibility of Concourse enables us to continuously test and deploy our Pivotal Cloud Foundry Services, and with minimal setup, add a Mac Mini worker for on-device mobile testing.

Manually Provisioning An OS X Worker

To begin running mobile tests on a Concourse pipeline, you first need to set up a worker on an OS X machine. For our purposes, we’ve set this worker up on a Mac Mini with a USB hub to expand the number of devices that can be used for running tests in parallel. The Concourse Docs describe how to manually provision workers on custom hardware without using BOSH. This process involves installing and setting up Houdini, a no-op Garden backend for OS X and Windows, and a SSH forward between Houdini and your Concourse cluster.

Keeping It All Running Smoothly

Once you’ve installed everything, and tested running Houdini and the SSH tunnel from a user shell, there are a few steps to follow to keep everything running smoothly. Since the houdini and ssh processes aren’t managed by BOSH, we need to daemonize these tasks and restart them if they happen to crash or lose connection to Concourse. We’ve opted to create Bash scripts which run them, and add them as agents using OS X’s launchctl command to keep them running.

Houdini Script

Because OS X launch agents don’t run in a user shell, we need to define some environment variables for Houdini to launch both XCodebuild and Android gradlew tests, while retaining a sane PATH setup. To eliminate a lot of the guesswork, while including all of the environment variables required to run XCode’s build tools from a Houdini container¹, you can echo the `env` of a logged in user’s shell to a ‘.houdini_profile’ ( eg. env > .houdini_profile ) and source that file before starting Houdini.

run-houdini.sh

#!/bin/sh
source /Users/pivotal/concourse_worker/.houdini_profile
houdini -depot=/Users/pivotal/concourse_worker/containers

Next we want to write the .plist file to be loaded by launchd, which will keep the script alive and run it at startup. I won’t go into too much detail about OS X Launch Daemons, as there are resources available on the Mac Developer Library which describe what each field does. In this case, we placed all of our plist files in the `/System/Library/LaunchAgents` folder to ensure they run at startup. Be sure to set the ‘Label’ key to something which is uniquely identifiable, and will be clear to anyone else managing the machine.

io.pivotal.concourse.houdini.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>AbandonProcessGroup</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>io.pivotal.concourse.houdini</string>
    <key>Nice</key>
    <integer>0</integer>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/pivotal/concourse_worker/run-houdini.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Advertising Worker To Concourse

Most of the process of forwarding a local Garden server is covered in the Concourse Docs, but writing Bash scripts and managing them as LaunchAgent .plist files makes these commands more robust by restarting them on boot, and whenever they crash. When run from a user shell, the SSH forward command will lose connection and stop running if there is any network disruption, so it is a good idea to run this command as a LaunchAgent as well.

The worker.json file is used to advertise your worker’s details over the SSH tunnel to the Concourse TSA. We’ve set our Mac Mini up as a ‘darwin’ platform worker with tags listing the supported devices.

worker.json

{"platform":"darwin","tags":[‘ios’, ‘android’],"resource_types":[]}

The next step is to create a Bash script which will run the ssh forward-worker command to tunnel the Houdini server to the Concourse TSA.

#!/bin/sh

ssh -vvv -p 2222 your.concourse-uri.com
-i /Users/pivotal/concourse_worker/mac_mini_worker_key
-R 0.0.0.0:0:127.0.0.1:7777
forward-worker < /Users/pivotal/concourse_worker/worker.json

forward-worker.sh

The .plist for running this script is the same deal as it was for Houdini, just be sure to set a unique filename, label, and point to the forward-worker.sh script.

io.pivotal.concourse.forward-worker.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>AbandonProcessGroup</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>io.pivotal.concourse.forward-worker</string>
    <key>Nice</key>
    <integer>0</integer>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/pivotal/concourse_worker/forward-worker.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
</dict>
</plist>

Pooling iOS Test Devices

For our on-device iOS app testing, we use ios-deploy to install the app, and Calabash-ios to run the tests. We have multiple iOS devices connected to the Mac Mini, and want to ensure that devices can only be used by one job at a time. Concourse’s pool resource allows us to model locks on the devices, and integrate them with our Concourse jobs to avoid messing around with them mid-test.
.
├── ios-devices
│ ├── claimed
│ └── unclaimed
│ ├── a0819
│ └── a1370
├── pool-party.gif

Inside the device pool repo we created plaintext files for each iOS device attached to the Mac Mini, which contain details required to target them for testing. We also have a GIF of a bear jumping into a pool because it’s funny.

Our iOS device lock files contain the following parameters:

Running Tests on Device

Now we’re ready to bring everything together in a script which will be run by our Concourse job. We include a `run-tests` script which contains the commands to build the app and run tests on device in a repo along with all of our CalaBash tests.

Here’s a brief rundown of our `run-tests` script:

Once we’ve sourced the metadata from the iOS pool and done some test data setup, we run xcodebuild and pass in the PROFILE_ID parameter for the device we’ve acquired a lock on:

xcodebuild [...] clean build PROVISIONING_PROFILE=$PROFILE_ID

Once we’ve built the app for that device’s profile, we can use ios-deploy to install the app onto the device using the –id flag.

ios-deploy --bundle $(dirname $0)/app_name.app --id $DEVICE_UDID

Finally, to run the tests on the device, we call cucumber with a few flags for the target device and its endpoint, along with the bundle id of the app.

BUNDLE_ID=io.pivotal.io.app_name DEVICE_TARGET=$DEVICE_UDID DEVICE_ENDPOINT=http://$DEVICE_IP:37265 cucumber

Here’s a trimmed down example of how our job uses the ios-devices pool resource to run the tests on device:

- name: ios-sample-test
  plan:
    - put: ios-devices
      params:
        acquire: true
    - put: env
      resource: acceptance-env
      params:
        acquire: true
    - get: ios-app-tests
      trigger: true
    - get: ios-app-package
      trigger: true
    - task: test-on-device
      config:
        platform: darwin
        run:
          path: /bin/Bash
          args: ["-c","source ios-devices/metadata && env/metadata && cd ./ios-app-tests/scripts/run-tests"]
        inputs:
        - name: ios-app-package
        - name: ios-devices
    ensure:
    aggregate:
      - put: ios-devices
        params:
          release: ios-devices
      - put: acceptance-env
            params:
              release: env

Now that we’ve got a pipeline task setup to run our tests on device, lets kick off a build:

Because we’ve set up all of the iOS code repos as triggers, the job will run any time a new build of the code is created.

When this task runs, Concourse will:

1. Acquire a lock on an available ‘device’ metadata file out of the ios-device pool
2. Acquire a lock on an available acceptance environment where the backend for the app is deployed
3. Download our app package from S3
4. Clone the acceptance tests from Github
5. Source the metadata about the device and test environment
6. Run the `run-test` script on the Mac Mini on the device against our test environment
7. Release the locks on the claimed device and environment

Here’s a look at a device picked out of the pool by the Concourse job, which runs through Calabash tests:

Next Steps

Now that we have set up a pipeline to run tests on devices, we can work on our iOS codebase and iterate rapidly, knowing that our tests are being continuously run on devices. From a product standpoint, Concourse’s pipeline definition semantics let us pass along the latest build of our iOS app to other jobs which can perform more tests, or promote a passing build to a release of the product.

About the Author

Biography