Immutable Infrastructure, Windows, and BOSH

August 24, 2016 Jared Ruckle


sfeatured-36436-WindowsIT has always been, in part, focused on a quest to squash variability in systems—to eliminate discrepancies wherever they may be. The more uniform an environment, the easier it is to discover and diagnose problems.

So when it comes time to make a change (i.e., patch an operating system, upgrade a piece of middleware, update an app), the goal is to complete the job as quickly as possible, and re-introduce the desired homogeneous state.

Over the years, a common toolset of RDP sessions, wizards, Domain Join, and System Center have emerged in an attempt to help Windows administrators to accelerate some configuration tasks. As a result, it’s now much easier to perform bulk updates and apply policies to collections of servers with just a few clicks.

Yet, environment consistency is still easier said than done. Chad Fowler describes why:

Because an old system inevitably grows warts. They start as one-time hacks during outages. A quick edit to a config file saves the day. “We’ll put it back into Chef later,” we say, as we finally head off to sleep after a marathon fire fighting session.

Cron jobs spring up in unexpected places, running obscure but critical functions that only one person knows about. Application code is deployed outside of the normal straight-from-source-control process.

The system becomes finicky. It only accepts deploys in a certain manual way. The init scripts no longer work unless you do something special and unexpected.

And, of course the operating system has been patched again and again (in the best case) in line with the standard operating procedures, and the inevitable entropy sets in. Or, worse, it has never been patched and now you’re too afraid of what would happen if you try.

The system becomes a house of cards. You fear any change and you fear replacing it since you don’t know everything about how it works.

Nearly every enterprise has old systems, similar to those Fowler describes above, and suffers the same pain.

Now here’s the good news: New tooling has come along to solve this problem. Many vendors have rallied around an idea that promises to transform the nature of infrastructure management—immutable infrastructure.

Immutable Infrastructure: The Low Down

Immutable infrastructure is the idea that once a VM has been created, it should never be modified.

Josh Stella describes why you should stop managing infrastructure and start really programming it:

“Immutable infrastructure (II) provides stability, efficiency, and fidelity to your applications through automation and the use of successful patterns from programming…the basic idea is that you create and operate your infrastructure using the programming concept of immutability: once you instantiate something, you never change it. Instead, you replace it with another instance to make changes or ensure proper behavior.”

Chad Fowler,who coined the term, further explains the concept:

“If you absolutely know a system has been created via automation and never changed since the moment of creation, most of the problems…[with infrastructure variation] disappear. Need to upgrade? No problem. Build a new, upgraded system and throw the old one away. New app revision? Same thing. Build a server (or image) with a new revision and throw away the old ones.”

This may sound similar to another idea transforming infrastructure management: thinking of your servers as cattle, not pets. CERN’s Data Centre Evolution presentation captures this notion well:

  • Pets are given names
  • Pets are unique, lovingly hand raised and cared for
  • When they get ill, you nurse them back to health

By contrast:

  • Cattle are given numbers
  • They are almost identical to other cattle
  • When they get ill, you get another one

Immutability is the logical extension of the cattle idea, and it’s vital to building software-defined businesses. That’s why Pivotal created BOSH: an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.

BOSH has long helped Linux sysadmins achieve infrastructure immutability—and Windows administrators will soon be able to leverage BOSH to automate infrastructure management at scale.

Let’s Get Real: 6 Ways BOSH Will Improve Infrastructure Management For Windows

So how does BOSH actually work for a Windows shop? Let’s go through a few real-world scenarios.

1) Installing Software
BOSH uses a concept called releases to distribute software. In this context, releases include all of the required components to install a given piece of software. This includes the binaries, configuration properties, and startup scripts—everything that’s needed for reliable, consistent deployment. There’s no need for a VM to reach out to a package server at deploy time. That’s a big deal, and a big step on the way to immutability.

Pivotal uses releases in combination with BOSH’s deployment manifests to create an accurate, precise map of all of the states of an environment. This allows us to easily stand up a pre-prod environment that is identical to our production environment, for example.

2) Updating Software
Unlike many tools in this space such as Chef, Puppet, or Ansible, BOSH is built specifically to support rolling updates of software—meaning no downtime during the process. This continuous uptime is achieved through concepts like canary updates and “drain” (where the active jobs of the VM are gradually wound down). With BOSH, upgrades to production environments are routine—a non-event that users don’t even notice. In fact, Pivotal’s public hosted version of Pivotal Cloud Foundry, Pivotal Web Services, uses this model to update its infrastructure and base VMs weekly, and no one notices.

3) Pushing Windows Updates
Instead of pushing updates to VMs, as is done with Domain Join, Pivotal engineers bake any new security updates into the underlying operating system image that runs our software, known in BOSH as a stemcell. This allows release creators to certify that their software runs as expected with the newest patches. For us, the workflow is easy as changing the stemcell version in our manifest, and running `bosh deploy`. The security updates will roll through the system just like a release update—new, patched machines will be deployed, and old, unpatched machines will slowly transition their workloads over and disappear.

4) Group Policy
Similarly to how updates propagate, Pivotal engineers harden the stemcell with best practice group policies. We’re also free to apply other group policies using a concept known as BOSH addons. Addons are long running jobs or short lived tasks that can be used to apply configuration to any VM that BOSH creates.

5) Antivirus Updates
BOSH addons can also be used to bundle up an antivirus agent and its latest updates. To apply antivirus updates to every machine in our cluster, it’s as easy as `bosh deploy`.

6) Security—Rotate, Repave, and Repair
In addition to the above advantages for an immutable infrastructure approach, we at Pivotal believe the key differentiator is the security model that it unlocks. Justin Smith states:

“Its idea is quite simple. Rotate datacenter credentials every few minutes or hours. Repave every server and application in the datacenter every few hours from a known good state. Repair vulnerable operating systems and application stacks consistently within hours of patch availability. Faster is safer. It’s not a fantasy — the tools exist to make most of this a reality today. Do it, and you’ll see a dramatic improvement in enterprise security posture.”

By moving to an immutable infrastructure model, we can treat many VMs in a data center as ephemeral objects that can receive and respond to messages. They can be destroyed and recreated at will, instead of long lived pets that must be protected. Thus, we can then re-create VMs in a rolling fashion every few hours, making it extremely hard for a malicious actor to gain a foothold in an environment. Without immutable infrastructure, we cannot destroy and recreate VMs with the same confidence. In addition, we can easily enable canary-style deployments that allow us to roll out the latest patches, and AV updates to a few hosts to start with. And if the deployment on that limited scale goes well, the remaining servers can be safely updated.

What’s more, IT pros gain the confidence to make these changes during normal business hours, without disrupting the enterprise. That means no more maintenance windows in the wee hours of the night. WIth immutable infrastructure, IT gets a little less chaotic, and allows staff to focus on more transformative projects.

BOSH Windows—Where Are We Going Next?

We plan on shipping a beta of BOSH Windows in 2H 2016, and then opening the early access to a larger audience after collecting initial feedback. Contact us to request early access.

Want To Learn More About BOSH?

We recommend these links:


About the Author

Jared Ruckle

Jared works in product marketing at VMware.

Follow on Twitter Follow on Linkedin More Content by Jared Ruckle
How Far We’ve Come
How Far We’ve Come

Over the past 35 years, I have been fortunate enough to have had a ringside seat at three major transitions...

Two Subtle Influencers in Product Design
Two Subtle Influencers in Product Design

A discussion of a few factors that can influence users and affect product design.

SpringOne at VMware Explore 2023

Learn More