Today, the Linux Foundation announced a new foundation: the Continuous Delivery Foundation (CDF). For our part at Pivotal, we’re excited to see our partner CloudBees contribute Jenkins to the CDF. Netflix and Google’s Spinnaker—a project we continue to contribute to within Pivotal R&D—is also part of the foundation.
The creation of CDF highlights a category of tech that’s crucial to any big business. We like to say that continuous delivery is the practice of finding the balance between reliability and innovation. When you do continuous delivery right, you can safely allow changes to make it into the hands of users as quickly as possible. You become a continuously delivered enterprise.
To achieve this outcome, two things are essential: the right tools and the right culture. Let’s start with the easy part: tools.
With the Right Tools, Your Cost to Deploy Code Goes to Zero
Let’s be clear on the terms in play. An excellent white paper describes our key concepts of CI/CD as follows:
Continuous Integration (CI) is the process of automatically building software after new bits of code are integrated into a shared repository. This yields “builds” of the code base that are in a working state at all times.
Continuous Delivery (CD) is the process of automatically deploying the artifacts created with CI. [Tools]...are used to provide a consistent method of deploying software and configuration changes into an environment. The consistency CD provides means production deployments become a non-event.
Your CI tool takes new bits of your custom code, and constantly tests (i.e., “integrates”) them. When your team is ready to take that code to production, you enter the continuous delivery phase. The CD tool takes over, and deploys your code in a safe, secure way. Combine these two together, and you have an automated path to production.
We’ve invested in two open source projects to help organizations achieve this: Concourse and Spinnaker.
For Continuous Integration, Use Concourse
Concourse is an open-source continuous integration project. It helps development teams with these modern practices:
-
Test-driven development with extensive automated testing
-
Multiple code versions that must maintain compatibility
-
Many code derivatives, such as multiple IaaS targets or configurations
-
High-velocity teams that deliver code frequently
Since we launched Concourse, we’ve seen it take off at Cerner, Zipcar, the US Air Force, and others.
Another reason why Concourse is so popular: it makes automated build pipelines a first-class citizen. And it visualizes them beautifully; everyone knows the status of the build at-a-glance:
A typical Concourse dashboard, from "Designing a Dashboard for Concourse"
Concourse is just one part of your path to production. Spinnaker is the other.
For Continuous Delivery, Use Spinnaker
Spinnaker is a continuous delivery tool created by Netflix and Google. It encapsulates years of experience developing and delivering modern microservices at scale at Netflix.
Spinnaker is an excellent fit for the continuously delivered enterprise. It's multi-cloud and multi-runtime. It runs at scale. You can manage all your deployments through a single instance of Spinnaker.
As you get better at software, your continuous delivery complexity goes up. That’s where Spinnaker really shines. It gives you visibility into what you have deployed through application inventory and monitoring systems integrations. What’s more, Spinnaker makes it easier for you to adopt modern development techniques.
Want to move to blue/green deployments and rollbacks? Spinnaker has you covered. The same goes for automated canary analysis. Thanks to the hard work of many people over many years, you don’t have to figure all of this out on your own.
Why is Spinnaker so powerful? If you guessed “the open-source community” you’re right! The Spinnaker community is “dedicated to making deployment pain go away.” (That’s a quote from the Spinnaker book; it’s worth a read.)
Given all this, Spinnaker is a natural complement to Cloud Foundry. So we’re working with the community to bring a Cloud Foundry integration to Spinnaker:
Great updates from @pivotal @cloudfoundry team around @spinnakerio
— R Arni (@r_arni) December 20, 2018
Great new driver and docs..https://t.co/0pcvrthwg8
Also awesome to see @springcloud pipelines adopt @spinnaker https://t.co/LOYzANklAj ! @springboot +@concourseci + @cloudfoundry + @spinnakerio !! all in one!!! pic.twitter.com/hOz7DHEGWp
Spinnaker is at its best when its implemented by app developers for app developers. Here’s an excellent talk that showcases this use case in action:
"Spinnaker and the Distributed Monorepo" https://t.co/JPcQQxUOPR @jon_k_schneider <--IMHO the whole @kubernetesio community needs to study app CD primitives in Spinnaker to get app perspective
— James Watters (@wattersjames) February 20, 2019
Of course, we realize that every organization is different. If you prefer public APIs, or solutions from CloudBees, more power to you. The important thing is to have tools you’re happy with, and a culture that helps you capitalize on the promise of CI/CD.
Culture Is the Other Part of the Continuously Delivered Enterprise.
Yes, you need tools to deliver a frictionless path to production. But you also need a culture that supports this concept. We spend a lot of time with large organizations to help them do just that.
Let’s go back in time a few decades.
Be Proud of Your Monolith. But Don’t Be Happy With It.
We all remember the days of releasing a monolithic application infrequently. Many of us recall the pain of maintaining a vast landscape of servers. Those were the days of battling configuration drift, security patching nightmares, and never-ending software upgrades.
The building and running monoliths were Herculean efforts. Scores of people gathered all the application dependencies and packaged them together. Dozens more would validate the whole thing in a test environment with an extensive suite of integration tests. With a bit of luck, the collective team would actually ship something at the end of it all.
With the adoption of elastic cloud services, much of the infrastructure complexity can be handled at a platform level (with Cloud Foundry), or at the orchestrator level (with Kubernetes). But that complexity hasn’t gone away. It merely shape-shifted to the app layer. Your application is not one piece of deployable software. It’s many services that must work together to deliver a great user experience.
What does this new challenge look like? It gets a lot easier to sort through if you embrace on key truth: replicating production and development environments is impossible.
You Don’t Know What’s Going to Break Until You Go to Prod
The sheer amount of moving parts in the system of your application cannot be replicated with high confidence in your development environment. You need to shift your mindset shift from “things break when I go to production” to “I will never know what is going to break until I go to production”. The key is to de-risk your delivery pipeline by controlling your exposure. Use progressive rollout and inflict purposeful chaos in your systems. This tests resiliency, instead of just relying on consistency.
“De-risking your delivery pipeline” is easier said than done. So how are your peers meeting this challenge? For starters, they realize that delivering modern applications means a new way of thinking about what an application is. You should also think differently about your infrastructure and your delivery methodology.
Shift Your Mindset with These Three Changes
Here are a few important mind-shifts that you need to make when approaching cloud-native delivery:
Change How You Think About Delivery Risk
When you deliver parts of your applications separately, often as microservices, you need to mitigate risk in new ways. You should stage the updates and new features to minimize impact. New patterns like feature flags, A/B testing, and canary testing are now common, so you have a path to your software-defined future.
What if a microservice has inadvertently caused a problem elsewhere? No big deal. Just rollback seamlessly, there are plenty of workflows to choose from. Worried about an explosion of new dependencies and backing services? Don’t be. It’s easy to manage when they are brokered at the application level through clean APIs and contract testing.
Change How You Think About, Well, Change
Cloud-native delivery is about continuously delivering any change to your modern application stack —whether that is a new feature, a new service, a database upgrade, a bug fix, or security patch. You need a frictionless way to update your platform and the custom code that runs atop said platform.
This broad view of what constitutes a software update is why we see the workflows of platform delivery and custom application delivery converging. These days, app teams and platform teams use the same tools and the same practices. There’s no reason to manage these delivery paths differently, even if they operate at different layers of the stack. Both require testing, governance and zero-downtime deployments.
Change How You Think About Who Does What
Instead of someone who manages a locked-down delivery pipeline, think about enabling app product teams. Move away from a gatekeeper model, in favor of independent teams empowered to build and maintain their own path to production. With a focused engineering team to provide accelerators, reference examples, and consultation to help each team get past impediments.
Another concept to explore in your org is site reliability engineering (SRE). SRE is a new practice that helps to bridge the dev and ops gap through the lens of the application itself. SRE teams monitor and analyze behavior across the app portfolio to identify opportunities to “level up” the performance of the application state. It also provides a pragmatic, data-driven framework for app teams to continue to improve.
How to Make It All Work
So you’ve hit refresh on your mindset. How do you actually make cloud-native continuous delivery work? Each company has some unique nuances to their journey. But these three practices are consistent in every company that’s great at software:
-
Automate everything...from code commit to production deployment to monitoring and feedback. But value stream map it first! Kicking off automation can actually change your process. (Lifting and shifting an old process into automation rarely works.)
-
Everything as code should be your mantra. Whether its pipeline, infrastructure, database or application code, version-controlled assets fuel your automation.
-
Progressive delivery works the application to the point of providing the best experience for your customers. At it’s best, this path is iterative. And you should expect failure, and continuous learning along the way.
If you’re ready to explore what a changing mindset on software delivery might mean for your organization, Pivotal can help. Through a Pivotal Labs engagement, we can take you through a whiteboard session where apps and ops teams come together. We’ll help you identify what parts of your application portfolio or process need improvement.
We’ll also help you transform those monolithic applications and associated delivery processes, so you can run them with greater agility atop a cloud-native platform!
Want to hear more about what we’re up to with CI/CD? Head to CF Summit next month, where it’ll be a hot topic.
You should also check out the webinar “Continuous Delivery to the Cloud: Automate Thru Production with CI + Spinnaker.
About the Author
