Delivering Container Security in Complex Kubernetes Environments

February 9, 2021 Brad Bock

You may have noticed the VMware Tanzu team talking and writing a lot about container security lately, which is no accident. As DevOps and Kubernetes adoption continue their exponential growth in the enterprise, securing container workloads consistently is among the most difficult challenges associated with that transformation. There is a term we have been seeing—and using—a lot lately that encompasses a new way of looking at container security for Kubernetes: DevSecOps. 

In this post, we will walk through some of the key DevSecOps practices that can improve your container security and compliance posture—even as the number of containers you deploy drastically increases.

The journey from DevOps to DevSecOps 

The term DevOps was coined alongside the widespread adoption of configuration management software, tools that automate many of the processes involved in shipping updates to an application in a virtual machine (VM) or bare metal environment. The advent of such automation enabled new, agile methods of building, shipping, and deploying software, which, in turn, reduced friction between developers and operators. And because all of these applications were built in-house, it was possible to track which dependencies and operating system libraries were present in the infrastructure, though whether that tracking was actually done varied widely between organizations. 

When Kubernetes arrived, DevOps became less a choice than a necessity. After all, containers cannot be logged into and updated the way applications in VMs or bare metal servers can. Rather, updates to existing code, dependencies, or operating system libraries are built into a new image, which is then deployed in place of the existing container. Because each application is composed of multiple containers running as microservices, maintaining applications through manual processes quickly becomes impossible.

Today, most custom software intended to run on Kubernetes is built into containers by the developers who created it. This work is done in parallel by individual developers and teams, on separate release cadences, using processes that are opaque to anyone inspecting those containers later on. All of the open source software that teams used to install in VMs and bare metal servers are now consumed as pre-packaged containers obtained from myriad repositories across the internet—typically without any readily available information about what’s in the stack or how it was configured. This is great for speed and agility, and presents a best-case scenario for developers who need to move fast. For security teams, however, having thousands of potentially vulnerable containers built on images with unknown provenance lurking in development environments and occasionally finding their way to production is unacceptable.

DevOps vs DevSecOps

While the term DevSecOps can be controversial—some consider the addition of “Sec” to DevOps unnecessary—it has helped frame the cultural and technological transformations that must happen across the entire IT industry if we are ever to see a world where catastrophic security breaches are not a regular occurrence.

Among the most important DevSecOps practices you can undertake in a Kubernetes context is to create policies requiring that only approved software is allowed to exist in your infrastructure and enforcing those policies in an automated way. Doing so requires a complete and transparent accounting of all the code, binaries, and operating system libraries in every container you build or consume. It also requires implementing a set of configuration policies that apply to every base operating system image underlying every container so as to make it clear what is, and isn’t, compliant. 

Achieving that level of transparency can be a monumental undertaking, one that requires dedicated teams of developers and application operators who are experts in the intricacies of application packaging. Many teams view it through a pie-in-the-sky lens as something they aspire to achieve, but have no idea where to begin. There is hope, however, as the Kubernetes community and vendors like ourselves are increasingly weaving DevSecOps into the fabric of the platforms, products, and services we build.  

Integrating security in DevOps

Solving the security and compliance challenges inherent in adopting Kubernetes is typically done in one of two ways: By either setting up a dedicated team that does nothing but package golden OS images and open source applications, or by having developers and operators maintain their container images themselves. But because that work is difficult, tedious, time-consuming, and expensive in practice, there can be pressure for teams to circumvent security altogether.

What is needed is a completely new approach to building custom application containers and consuming off-the-shelf open source software. With VMware Tanzu Advanced, that is exactly what you get, out of the box. 

How Tanzu Advanced makes container security intrinsic to DevOps 

Tanzu Advanced comes with two capabilities that make successful DevSecOps possible from day one. The first is an automated container build system; the second is a catalog of the most commonly used open source containers and Helm charts. 

Automated container build system

The automated container build system builds containers from source code on pre-approved base OS images, burning a detailed manifest of each container’s contents by default. The build system can then automatically rebuild and push new images whenever application code, dependencies, or OS libraries are updated. 

This function is managed centrally, with no intervention from individual developers required; operations or security teams can patch vulnerable libraries or dependencies in every container independently of application code or binaries. For any application built using Tanzu Advanced edition’s automated container build system, the combination of centrally managed updates of vulnerable or out-of-date containers with accurate information about every container’s contents enables the security compliance process to be automated.

A continuous stream of image updates for every container.

Open source application building blocks

The second capability, a catalog of the most commonly used open source containers and Helm charts, is also packaged on a pre-approved base operating system image. This library of open source application building blocks is kept up to date with an industry-leading pipeline. It rebuilds the entire catalog with the freshest OS base image daily as well as any time an individual app update or vulnerability patch becomes available. Much like the automated build service for custom app containers, each container in the catalog comes with a complete and verifiable accounting of all its contents that is accessible via CLI or offline through an OCI registry. For an extra level of confidence, the containers are also tested and scanned for vulnerabilities before updates are pushed to the repository. This combination of fast updates and accurate information about every container’s contents enables the complete automation of security compliance for any application sourced from the container catalog.

A complete accounting of every container’s contents enables security automation.

Together the automated build system and container catalog yield outcomes on par with or superior to what teams would get by maintaining their images themselves, minus the expense and potential for human error. Instead, developers are free to focus on their code and to easily self-serve open source building blocks already pre-approved for use in production. Operations teams can set up a continuous delivery pipeline that feeds a constant stream of image updates to production environments. And continuous compliance automation means that information about what is in every container can be consumed automatically, enabling security teams to implement invisible guardrails. Everyone moves at their own rapid pace, without compromising agility or security, to accomplish true DevSecOps.

Dive deep on DevSecOps with Tanzu Advanced

The container build system and app catalog described in this post are just two of the many ways that Tanzu Advanced delivers a huge leap in DevOps security. Check out these other great resources to learn more about the VMware Tanzu team’s vision for DevSecOps and a secure software supply chain: 

As always, we are eager to hear from you. So contact your friendly VMware representative for more resources and additional information, or let us know what you think on Twitter or Facebook.

Embracing DevSecOps for Modern Apps | VMware Tanzu at GitLab Commit
Embracing DevSecOps for Modern Apps | VMware Tanzu at GitLab Commit

Next Video
Security as Code: A DevSecOps Approach
Security as Code: A DevSecOps Approach

Security as Code (SaC) is the methodology of codifying security tests, scans, and policies. Security is imp...