This is the first in a series of blog posts looking at different ways to deploy software on Pivotal Container Service (PKS). In this post we’ll focus on how to do continuous delivery of your software on PKS (or any Kubernetes cluster) using Spinnaker.
Pivotal Container Service is a platform built by Pivotal to ease the burden of deploying and operating Kubernetes clusters. PKS builds on top of Cloud Foundry’s container runtime (formerly Kubo) that utilizes BOSH to handle both Day 1 and Day 2 operations for Kubernetes.
PKS, however, is not just another Kubernetes distribution. It’s an on-demand service that deploys and manages Kubernetes clusters, much like services from public cloud providers like Google Container Engine (GKE). This allows the Kubernetes operator to customize their Kubernetes usage to match their business requirements much more closely than most on premises Kubernetes deployments by reducing the operations overhead and daily toil required to manage the clusters.
PKS integrates easily with VMWare, giving you a GKE-like experience inside your own datacenter and removing the toil of building and installing servers (physical or virtual) by managing the entire deployment including orchestrating the necessary VMs with VMWare.
Introducing Spinnaker
Spinnaker is an enterprise-grade continuous delivery tool originally built by Netflix to deploy Netflix applications as is part of the excellent suite of Netflix OSS projects. It is is an open source, multi-cloud continuous delivery platform that helps developers release software changes with high velocity and confidence.
Spinnaker treats cloud-native deployment strategies as first class constructs, handling the underlying orchestration such as verifying health checks, disabling old server groups and enabling new server groups. Spinnaker supports the red/black (a.k.a. blue/green) strategy, with rolling red/black and canary strategies in active development.
My PKS Lab
I am running Pivotal Container Service on Google Compute Engine and have used the pks cli to create three clusters: One cluster running Spinnaker called `cicd`, another small cluster runningjust a single k8s master for dev, and third larger HA cluster for prod.
Deploying Spinnaker
Before using Spinnaker with Kubernetes I need to deploy it. Spinnaker comes with a tool called Halyard to help with installation. However, they have some quickstart options for people who just want to kick the tires. I chose to use the Helm chart from the quickstart guide.
In the Spinnaker config (via Helm variables) I was able to configure access/credentials to my Docker registry, Git repo, and Kubernetes clusters. Using Helm it was incredibly easy to install:
$ helm install --values ../values.yaml --namespace spinnaker -n cicd . NAME: cicd LAST DEPLOYED: Wed Jul 11 10:04:37 2018 NAMESPACE: spinnaker STATUS: DEPLOYED … … NOTES: 1. You will need to create 2 port forwarding tunnels in order to access the Spinnaker UI: export DECK_POD=$(kubectl get pods --namespace spinnaker -l "component=deck,app=cicd-spinnaker" -o jsonpath="{.items[0].metadata.name}") kubectl port-forward --namespace spinnaker $DECK_POD 9000 2. Visit the Spinnaker UI by opening your browser to: http://127.0.0.1:9000 For more info on the Kubernetes integration for Spinnaker, visit: http://www.spinnaker.io/docs/kubernetes-source-to-prod
After a few minutes I could see that all the pods were up and running and ready to use:
$ kubectl get pods NAME READY STATUS RESTARTS AGE cicd-redis-d9bb99f4c-79rh9 1/1 Running 0 4m cicd-spinnaker-clouddriver-7f87cfbd5d-gxg74 1/1 Running 0 4m cicd-spinnaker-deck-7c8898fb57-tfwcg 1/1 Running 0 4m cicd-spinnaker-echo-779784b654-fd7tj 1/1 Running 0 4m cicd-spinnaker-front50-656b764fc9-hn2b9 1/1 Running 0 4m cicd-spinnaker-gate-d96459c4-hmjt7 1/1 Running 0 4m cicd-spinnaker-igor-6b67f78497-6hfk7 1/1 Running 0 4m cicd-spinnaker-orca-7547fd5c4c-pwfg2 1/1 Running 0 4m cicd-spinnaker-rosco-fc684f57c-tbp6s 1/1 Running 0 4m $ chart k port-forward cicd-spinnaker-deck-7c8898fb57-tfwcg 9000:9000 Forwarding from 127.0.0.1:9000 -> 9000
Hello World
Of course, we also need an application to deploy, so like all good demos we’ll pick an example application that in no way resembles a real world application, a hello world app written in Go. As long as your application has a Dockerfile and you know what port it listens on, you can pick any application. Obviously, if you have a database backend you have a few more considerations around the deployment pipelines to consider.
We also need to set up automated builds for our image. While it’s possible to use Spinnaker to build images, the documentation suggests using the automated build tooling you have available with your Docker Registry. So I went ahead and created an automated build for this repository on Docker Hub and set it to not only build on master, but also on Tag creation. This way we can use Tags as a release mechanism in Spinnaker.
Deploy Hello World with Spinnaker
1. Create a Spinnaker Application
Browse to Spinnaker using the port forward we created earlier. We’re going to an application in Spinnaker. An application is a group of resources that Spinnaker manges in your cloud provider (in this case Kubernetes). If you have multiple applications you can group them into projects, but we’ll keep it simple for now and skip projects.
Click Applications in the top menu of the Spinnaker UI. You’ll notice there’s some applications already running. That’s because Spinnaker will show any running pods (or other Kubernetes resources) as applications.
Under Actions on the top right hand corner of the window click Create Application.
Give the Application a name and set the Repo Type up to be github and set the Repo Project to your github username and the Repo Name to the name of the repo. Set Instance Port to 80 and hit Create:
Once created you’ll be dropped into the clusters section for the new application, which should appear to be empty. A cluster is a collection of server groups that are running the same application at various versions. A Server Group is an application running in Kubernetes, either a ReplicaSet or a Deployment.
2. Create Server Group
Click the + button on the top right hand corner to create our first Server Group:
From here there’s quite a lot of configuration options. Thankfully, we don’t need to worry about most of them for such a simple application. Under Basic Settings we need to set the Account (Kubernetes Cluster) and Namespace (Kubernetes Namespace) that we want our application to run in. You’ll also give the Server Group a stack to help name the running version of the application and one or more containers to run that can be selected from the available Docker Registries and Repositories that you configured Spinnaker with:
Under the CONTAINER section you’ll need to specify the Container Port that your application will listen to inside the container:
Also in the CONTAINER section you’ll want to click Enable Readiness Probe and configure it to look at the same port:
Click Create and wait a few minutes for Spinnaker to deploy your application:
Once the Server Group is deployed you’ll get dropped back to the clusters screen:
At this stage Spinnaker has created some Kubernetes resources for your application. Let’s take a look at them using kubectl (make sure you configure your Kubernetes client for the correct cluster and namespace):
$ kubectl -n demo-dev get all NAME DESIRED CURRENT READY AGE rs/helloworld-dev-v000 1 1 1 8m NAME READY STATUS RESTARTS AGE po/helloworld-dev-v000-tgwrq 1/1 Running 0 8m
You’ll see that Spinnaker has created a ReplicaSet for your application which in turn has created a Pod. However there is no service configured, so you can’t access it via a NodePort or LoadBalancer. This is to be expected. We need to create Load Balancer resources in Spinnaker for that. But lets just use a kubectl port-forward for now to ensure our application is deployed correctly. By backgrounding the kubectl process we can run curl in the same terminal to validate our application is working:
$ chart kubectl port-forward helloworld-dev-v000-tgwrq 8080:8080 & [1] 2823 $ chart Forwarding from 127.0.0.1:8080 -> 8080 curl localhost:8080 Handling connection for 8080 <html><head><title>hello world</title></head><body>hello world!</body></html> $ chart kill %1 [1] + 2823 terminated kubectl port-forward helloworld-dev-v000-tgwrq 8080:8080
3. Create Load Balancers
Our application is pretty useless if we can’t access it. We can create Load Balancer resources in Spinnaker which manage Kubernetes Service resources. Go back to your Spinnaker UI and click the Load Balancers menu item in the top right hand corner.
We’re going to create two Load Balancers here, one for dev and one for prod.
Click Create Load Balancer (the + button):
Fill out the Account, Namespace, and Stack items under Basic Settings:
Under Ports set the Target Port to match your application port inside the container:
Under Advanced Settings set Type to LoadBalancer.
Hit the Create button and wait for the Load Balancer to be created (you actually don’t need to wait, you can close the progress window if you're impatient and move on to the next steps).
Create a second Load Balancer for Prod the same way:
Once created you should see both Load Balancers listed in Spinnaker. If you click one you’ll be able to see details such as the External IP associated with the Load Balancer:
4. Create Pipeline
Now that we have our Application and Load Balancing managed by Spinnaker we can create the pipeline that will deploy our app to development and allow us to promote it to production any time we create a new release in Github.
Click on Pipelines in the menu and then click on Configure a new pipeline:
Ensure the Type is set to Pipeline and the Pipeline Name is set to something meaningful such as “Deploy Hello World” and hit Create:
It will drop you to a configuration screen for the pipeline. Click on the Add Trigger button:
Set the Type to Docker Registry and then fill out Registry Name, Organization, and Image sections and hit Save Changes.
Scroll back up to the top and hit Add Stage:
Set Type to Deploy and give it a Stage Name of Deploy to Dev. Then hit Add server group:
Create a new Server Group from the dev Server Group created earlier and hit Use this template:
Under Basic Settings you just need to change the Containers, choose the “Images from Trigger(s)” option so that Spinnaker knows to pick the correct image based on the created release. Set Strategy to Highlander so it knows to delete any old versions of the application.
Under Load Balancers choose the helloworld-dev load balancer we created earlier:
Then Set the Container Port and create a Readiness Probe like we did earlier:
Save that stage and then immediately create a new stage:
Set Type to Manual Judgement and Stage Name to Promote to Prod. Add Promote to Prod? to Instructions. Hit Save Changes.
Create a New Stage:
Set it’s Type to Deploy and Stage Name to Deploy to Prod and hit Add Server Group:
Use our existing Dev Server Group as a template:
Set the Account and Namespace to match the Kubernetes cluster and namespace you wish to deploy to and then give it a Stack name of prod and change the Containers to the Tag resolved at runtime container. Set Strategy to Highlander:
Under Load Balancers choose your production load balancer, and under Replicas set Capacity to 3:
Like before change the Container port to 8080 and create a readiness check for port 8080:
Hit Add and it will take you back to the Pipeline configuration. Hit Save Changes:
Hit the Pipelines menu item again to get back to the Pipelines summary screen:
5. Create a release
Everything should now be set up to test the pipeline. First, let’s create a release for the current version of the application and watch it process through the pipeline. You do this in Github by first clicking on the releases text:
Click on the Create a new release button:
Give it a release version of 1.0.0 and a release name and hit Publish Release.
Once the release has been cut you’ll see Docker Hub queue up a new build:
After a minute or two it should switch to Building:
Before finally Success:
Once that has happened Spinnaker will kick in and execute Stage 1 of your new Pipeline:
Ignore the fact it says manual start, I forgot to take screenshots the first time around.
Since we set the second stage to require manual intervention it stops and waits for input from the user:
Before making a decision we can test our development deployment by clicking through on the dev Load Balancer Ingress IP:
Clicking on the IP should open a new tab in your browser and display your application:
This looks fine, so go back to the pipeline and respond Yes and the pipeline will progress to Stage 3. You’ll notice the progress bar is broken up by stage, so you can click on Stage 3 to see the progress of that specific stage:
A few moments will pass and the status should change to Completed:
Back to the Clusters page in Spinnaker you’ll see see a new Server Group called helloworld-prod. You’ll notice it shows as having three healthy pods as opposed to dev only having one.
You can click through the Load Balancer Ingress IP to test it and should get the same content as dev.
6. Update the Application and cut a new release
To update the application I create a new Pull Request in Github with some changes to the hello world text:
I then merged that PR and created a new release version 1.1.0.
Once again Docker will see the new tag and create a new build, which will in turn trigger an execution of the Spinnaker Pipeline:
Notice this time the execution shows it having been triggered by Docker since this time I remembered to take screenshots.
Spinnaker will prompt us to Promote to Production:
But before that we can check the dev environment out. You can see in the Clusters page that the dev Server Group has been updated with the new version, but the prod Server Group hasn’t:
As you’d expect the content behind the dev Load Balancer has been updated:
This all looks good, so we can tell Spinnaker to Promote to Production by clicking Yes to the Promote to Prod gate. This will allow our Pipeline execution to finish and deploy our app to Production:
Again, we can look at the Clusters page to see our Server Group has updated:
Finally, we can check the Load Balancer for the production Server Group and we’ll see that our app has updated itself there:
Conclusion
Overall, I’m quite impressed with how Spinnaker handles deploying applications to Kubernetes. It handles all the messy parts of creating the Kubernetes manifests and seems to be featureful enough to be able to deploy most types of applications.
Obviously, this was a very simple example application, but there are plenty of examples of Spinnaker being used to deploy complex microservice-based applications (like Netflix!), so I don’t doubt it can be used in the real world.
By relying on my Docker Registry to build images, it’s also asking me to rely on something upstream to do my unit testing and being able to plug something into the pipeline to be able to do integration testing. I know that Spinnaker is capable of creating AMI images for Amazon, so I would expect to see options for having Spinnaker perform the build and unit tests for Kubernetes as well.
Spinnaker itself has a number of pods and services to run it, enough so that I would be hesitant to use it for very simple applications. But, I can see how on larger teams with meaningful applications it could add a ton of value and help streamline the deployment processes.
About the Author