Case Study: How Pivotal Network Does Zero Downtime Deployment on Cloud Foundry

I’m a developer at Pivotal Labs and former technical lead for Pivotal Network. In this post, I’ll walk you through zero-downtime deploys to Cloud Foundry. I’ll use our experience deploying Pivotal Network as a case study.

At the end of last quarter we came out of a three week deploy freeze (sound familiar?), and we were itching to deliver features. We also wanted a way to avoid downtime constraints going forward. We established the steps I am about to outline, and subsequently deployed four times to our production environment in the span of that following week, with no downtime. We can now deploy small features as they are ready, during business hours. This a big boost to our confidence in delivering and moving quickly, and it means we’re not staying after hours doing late night deploys.

Deploying individual apps

Initially, deploy your code to two cf applications (if you have an existing application, create a second one). Then you can swap out one or the other as you please, and the Cloud Foundry router will automatically point to the one that is available!

Cloud Foundry will automatically bind whatever services you have defined via a manifest file or console.run.pivotal.io to each of the apps whenever you deploy.

The Cloud Foundry documentation on blue-green deployment describes this process in more detail.

Example – Rack App

We start with a pair of apps running the same code – a simple Rack app. This script assumes that both have been pushed with default routes created, and they share a common database service. They also share the route cf-blue-green.cfapps.io.

# config.ru
run Proc.new { |env| ['200', {'Content-Type' => 'text/html'}, ['Hello, world!']] }


# deploy.sh

#!/bin/bash
cf unmap-route blue-app cfapps.io -n cf-blue-green          # make the app unavailable to requests
cf push blue-app -c 'bundle exec rackup config.ru -p $PORT' # deploy the code changes to CF

# wait for the blue app to start
while true; do
RESP=`curl -sIL -w "%{http_code}" "blue-app.cfapps.io" -o /dev/null`
if [[ $RESP == "200" ]]
then break
else sleep 3 && echo "Waiting for 200 response"
fi
done

# make the blue app available to the router
cf map-route blue-app cfapps.io -n cf-blue-green

# deploy to the green app
cf unmap-route green-app cfapps.io -n cf-blue-green
cf push green-app -c 'bundle exec rackup config.ru -p $PORT'
cf app green-app
cf map-route green-app cfapps.io -n cf-blue-green

Github repo

Example 2 – Rails app

The same process above works with a Rails application, even when there are migrations—as long as the migrations are not destructive.

For example, if you write a migration to create a column, deploying the blue app can run that migration against the shared database without disrupting the green app. The green app’s code will not know about the column and will continue normally.

If we were deploying a Rails app, we’d take the same script from above, and make a minor adjustment to the command argument string:

cf push blue-app -c 'bundle exec rackup config.ru -p $PORT'

becomes:

cf push blue-app -c 'bundle exec rake db:migrate && bundle exec rails s -p $PORT'

Rolling back

In the event that the blue app fails to deploy, you can simply deploy the version of your code that was on the app prior to the deploy. Then you can have both instances still running while you debug the failure.

Conclusion

This is a simplified example, but even so it demonstrates the ease with which one can deploy to Cloud Foundry with zero downtime. Now that we’ve automated this process on Pivotal Network, we can deploy features several times a week, and execute hotfixes quickly with low overhead. This creates better feedback loops with our stakeholders and customers, and means we can regularly ship features without worrying about maintenance windows. It feels good!

About the Author

Biography