How VMware Tanzu was Impacted by the XZ Vulnerability: Spoiler Alert—It Wasn’t!

"You can observe a lot by watching." Yogi Berra

Many years after I left college and started my first “real” job managing telecom and IT in shipping, freight, and warehousing, I once again find myself immersed in supply chain security. The difference is that today my involvement includes the  foundation that businesses rely on build, deploy, and run applications.

The security of modern applications and their supply chains have been a concern for a long time, all the way back to the days of modern open source. There have been many reviews of the risks associated with using non-commercial code, and exploits of NPM, Solar Winds and other supply chain attacks have helped bring many of those concerns to fruition.

While there have been changes to the way businesses, and open source, build and release software, there are still risks to the supply chains we all depend on—most recently with the attack on xz.

There are many excellent write ups on the xz attack, like this great tldr or this one if you want a bit more technical detail, so I won't take our time here going into great detail or analysis of xz or the attack itself. Instead, I’m going to take you through the timeline and show how it impacted our products and our customers (spoiler alert: no VMware Tanzu product was impacted and no customers were at risk).

Zero CVE—More of a distraction than a solution

One thing we need to address when talking about supply chains is something that I think causes so many businesses angst and toil: the enterprises should work toward having “Zero CVE.” When looking at the world today, where everything (and I mean everything) is run by software and much of that software is dependent on open source  (“other people's code”), there is a slim chance that there will ever be zero CVEs. In fact, the base operating systems that make up the root file system of most containers have vulnerabilities that will never be addressed. So, be careful when building and depending on images that offer “zero CVE” because they only distract from what should be the focus: A proper, continuous lifecycle management for platforms, applications, and data.

Even moving toward distroless container-based applications, there are still always going to be vulnerabilities. And, even if a vulnerability isn’t high or critical, it is possible to chain lower level vulnerabilities into successful attacks on systems. That’s why focusing on trying to remove all CVE ignores the critically important governance of continuous patching and upgrading, as was originally highlighted by the 3Rs. 

The first thing to understand about a world built on open source is that not every project has the backing of a large enterprise. There are many little components that have a small team or possibly even only one maintainer. Unfortunately, the below diagram sums up the situation. 

^{[SOURCE: https://www.explainxkcd.com/wiki/index.php/2347:_Dependency]} 

These small projects can be critical and, as was pointed out during the discussions that have followed the xz attack, supporting a project for a long time can lead to burn out and ultimately result in opportunities for attacks like xz. 

VMware Tanzu currently supports more than 100 different open source projects, either through direct support by hiring developers or contributing to those projects in some way. A few of the most visible projects include Spring, Postgres, Cloud Foundry, Kubernetes, and the entire Bitnami catalog. However, we also depend on SSH through various operating system partners and through thousands of other open source dependencies, which created potential exposure to xz. 

In addition to contributing to security patches or notifications for Spring and Cloud Foundry, Tanzu also actively monitors open source, community, and other threat feeds,  which is where our joining of the xz attack occurred.

If we follow the timeline of the xz attack, we can see how the attacker spent almost a year pressuring the maintainer. This started with commits and attempts to “support” xz in October of 2021 while appearing to pressure, yet sympathize, with the stress the maintainer was under. Over time, this led to the attacker gaining commit rights to the repo in September 2022.

In June of 2023, an individual that does not appear to exist anywhere other than a June git commit and in 2024 to promote the back-doored xz. This commit didn’t provide the back-, but instead provided a hook by which the back door could modify the global function table before it is remapped as read-only.

With the hook ready for the back door, in July 2023, a commit was made to disable ifunc support during oss-fuzz builds, which by itself does not appear to be innocuous. It will be used in the later attack, so it is one more step in preparing for the attack.

In January of 2024, the web site for xz moved to github pages to gain more direct access to the web page that the attacker had control over (git access vs hosted on web server).

On February 24, 2024, the attack was merged with a binary back door commit to the binary test file folder. This folder was documented as having test files, so it was an attempt to take advantage of putting a binary file in a location that wouldn’t be carefully reviewed.

The next day the attacker merged the changes into a release with the extra build-to-host.m4 binary that adds the back-door when building deb/rpm package files. A more detailed review of the attack script can be found here.

Once the release was made available, automated pipelines started taking over from there. That same, Gentoo started seeing crashes in the xz release and, on March 4, Red Hat started seeing errors with the function where the backdoor started.

The attackers then started pushing for a change to libsystemd to remove libzma and get the errors removed before the OS vendors started digging into the errors in the build systems. The attacker committed fixes on March 5, which had been picked up by Debian testing by then.  The backdoor binaries were updated and committed by March 9, but it takes until March 27 for debian to update. By then the attacker was submitting bug fixes to Ubuntu to get them to pull in the compromised update.

The attack was discovered by Andres Freund on March 28 by “observing a few odd symptoms around liblzma … logins with ssh taking a lot of CPU.”  Debian and arch linux rolled back changes on the March 28, and Andres publicly shared the details on March 29.  By March 30, all releases had been backed out to the last secure release version.

Once it became public, Tanzu joined the rest of the world in searching for the malicious xz in any of our development or released products. The Tanzu incident response process kicked off immediately, coordinating an investigation across all product teams and all SBOM and application component supply chains, searching for anything that could have received the malicious xz release. 

The analysis across all Tanzu products was completed within hours, with the determination that  Tanzu only builds from stable, production-ready releases and does not use unstable or testing where there is more volatility. As a result, there are no Tanzu products that use the compromised xz releases. On March 30, Bitnami publicly announced that all bitnami images were not impacted.

• Tanzu Application Service and Tanzu Kubernetes Grid Integrated, jammy (and xenial) stemcells were not impacted: https://ubuntu.com/security/CVE-2024-3094 

• Tanzu Application Platform does not contain the vulnerable version of xz.

• Bitnami has debian but was not using any vulnerable versions: https://twitter.com/bitnami/status/1774019566143775079 

• Tanzu Application Catalog Photon, Ubuntu, RHEL, Debian and all ship non-vulnerable version of xz.

• Tanzu Data for bare metal, OpsMan/TAS tiles and Kubernetes distributions all ship with non-vulnerable version of xz.

• Tanzu SaaS, including CloudHealth, Observability, Guardrails, and Hub services are built on non-vulnerable distributions.

• Salt products use Photon, which is not vulnerable.

While the community responded rapidly and minimized the exposure of this one critical security attack, it has raised the bar once again for businesses and people to securely use and build products from open source. It also highlights the need for platforms, and the applications that run on them, to fully enable the rapid rollout of updates, regularly and often.

The table stakes security controls that the world has been working on, from zero trust to attestations of provenance like SLSA and a shift from manual to automated, reduce risks from mistakes or human elements and introduces the risk of supply chain risks being automatically bumped into multiple products. 

The challenges will continue, and so will our discussion around security and the risks of modern applications and platforms, so watch this space for more conversation and content about how to approach governance, risk, compliance and security for modern enterprise applications.

There’s a tool for that 

In the meantime, if you are working with containers or Kubernetes and are curious where you can start to understand open source dependencies and their impact on your security and compliance posture, we recently released the Tanzu Open Source Software Health Assessment tool. The tool can be used to analyze your Kubernetes clusters, local folders, or repositories with Helm charts, YAML files, and public container images stored in open repositories, such as DockerHub, or locally in private repositories.

After analyzing the resource you choose, the tool generates a report with details on all potential misconfigurations and security vulnerabilities that put you at risk, along with a list of applications from VMware Tanzu Application Catalog that could help you improve the security of your software supply chain.

More details can be found in this blog, and you can access the Tanzu Open Source Software Health Assessment here. 

Tanzu and the 3Rs 

One of the ideals promoted by Tanzu, and any modern application developer, follows the 3Rs mindset of releasing often. Tanzu products have regular updates and, as part of our Tanzu Security discussions, we will end every week with a security review of the week's releases.  

The summary below is a review of product release notes from a security and governance point of view, and may not reflect specific security or governance requirements you may require. It also does not include all details from the release notes. Please review the full release notes for product details that may have been missed.

Below you will find the security-focused updates to releases for every release so far this month:

Product updates 

April 6

Product updates: 

• Tanzu Data Solutions VMware Greenplum® - 6.27.0 - 2024-04-05

  • Minor release, no security updates

• SaaS Product Release Updates Tanzu Mission Control (TMC) - March updates

  • Added support for Kubernetes 1.29

  • Removed support for AKS Kubernetes 1.25 and 1.24

  • Removed support for EKS Kubernetes 1.24 and 1.23, 1.25 in extended support

• Spring Cloud Gateway for Kubernetes - 2.2.2 - 2024-04-05

  • Redundant Operator readiness probe has been removed

• Spring Cloud Gateway for VMware Tanzu - 2.0.8 - 2024-04-04

  • Resolved CVE-2023-52428 (nimbus-jose-jwt)

  • Resolved CVE-2024-22257 (spring-security-core)

April 19 

• Tanzu Application Service (TAS & TASW, ISO) - 6.0.1+LTS-T, 4.0.21+LTS-T, 5.0.11, 2.13.38 - 2024-04-17

  • Feature—Enable concurrent read/writes for HTTP/1

  • Version bump for many dependencies

  • tanzu-jammy-stack update

  • Buildpack updates

  • Xenial and jammy stemcell updates

  • FIPS compliant Jammy stemcells are available for Early Release. 

• Tanzu Kubernetes Grid Integrated—1.18.3 - 2024-04-16 

  • Bumped component versions

  • Added Strict-Transport-Securiy header to nginx HTTPS server on management console VM to improve scan results.

  • Upgrade failure resolved when special characters in vSphere password

  • Resolved ephemeral storage overrun with temporary logfiles from fluentd

  • SSH configuration improvements—CVE-2023-48795

  • Updated Samba to 4.18.4 to address CVE-2023-0922

• TKGi Management Console—v1.18.3 - 2024-04-16

  • Fixes issue of LDAP password appearing not hidden in MC-Generated TKGi configuration files

• Tanzu Data Solutions VMware Postgres for TAS - 1.1.2-build.6 - 2024-04-16

  • Postgres now supports S3-compatable backup

• Tanzu Data Hub (TDH)—1.0.0 - NEW GA - 2024-01-16

  • Initial release—No security updates

  • Security features:

    • Fleet management

    • Federated IDP

    • IAM and policy management

    • Log aggregation

    • RBAC

    • Auditing

• Tanzu Greenplum—5.30.0 - 2024-04-17

  • Tanzu Greenplum 5.30.0 contains no product changes. This release extends the End of General Support (EOGS) date to 2025-10-31.

• Tanzu GemFire—10.1.1 - 2024-04-17

  • includes updates to netty, spring, and spring-security to address the following security issues:

    • CVE-2024-22243

    • CVE-2024-22257

    • CVE-2024-22259

    • CVE-2024-29025

• Spring Cloud Gateway for Kubernetes—2.2.3, 2.1.10 - 2024-04-18

  • Resolved issue that prevented successful CORS validation of URL scheme on CircuitBreaker filter forwarded requests

• IPsec for VMware Tanzu—1.9.58 - 2024-04-16 

  • Maintenance release, no security updates

April 26 

• Tanzu data Solutions VMware Postgres—12.18.1, 13.14.1, 14.11.1, 15.6.1, 16.2.2 - 2024-04-15

  • This release introduces the advanced_password_check extension for checking passwords against password policies.

  • For Debian, this release adds a post-installation step that creates a default user account with permissions for the default database.

  • This release introduces the extension pg_failover_slots, which allows you to use logical replication slots across a physical failover in a physical streaming replication architecture.

• SaaS Product Release Updates: Tanzu Application Catalog (TAC)—Patch Release—April Updates

  • Added information about Pod Security Standards in Kubernetes.

  • Added information on listing container images from an OCI Hosted registry. For more information, see FAQs.

  • Updated instructions about SLSA level compliance since Tanzu Application Catalog is now generating SLSA Provenance attestations.

  • Added the definition of the Tanzu Application Catalog Build Type for SLSA Provenance attestations.

• Tanzu Mission Control (TMC)—Patch Release - 2024-04-25

  • Updated Velero—No security updates

• Single Sign-On for VMware Tanzu Application Service (SSO)—1.16.2 - 2024-04-25

  • Spring Boot dependency upgrade

Additional resources 

• Tanzu Defined: Strategy Update—Replay of Apr 3rd Event (Linked In, YouTube)

• VMware Tanzu Simplifies Accelerated App Delivery— - (Blog) General Manager Purnima Padmanabhan Shares Vision and Strategy for Tanzu Division.

• Purnima Padmanabhan and Ryan Morgan, Broadcom | Cube Conversation - (Video) Tanzu, as a separate division, is highlighted for its role in providing an app platform to accelerate app delivery, with a portfolio encompassing Cloud Foundry and Kubernetes-based solutions, data services and intelligence offerings. 

• Spring Webinar—Tanzu Spring Runtime (TSR) - Tanzu Spring Runtime brings you 24x7 Commercial support for the entire Spring ecosystem (Spring OSS projects, Apache Tomcat®, and access to extended support) from the experts who contribute and maintain the source. 

  • Apr 25 - Enhancing Compliance and Security with Tanzu Spring Runtime - Learn how you can utilize Tanzu Spring Consulting to provide your organization with hands-on support to help upgrade a portfolio of apps to the latest version of Spring, build new Spring apps, or migrate an existing application portfolio to Spring.

• Cloud Foundry Weeklies Series—Subscribe  | RSS Feed

  • Cloud Foundry Weekly Episode 8: Secure Cloud Foundry with Aqua Sec and DMZ - (Video) Cloud Foundry Weekly deep dives into various aspects of the Cloud Foundry ecosystem. Join Nick and Nicky as they interview special guests from the community and deep dive into AI Updates and Boosts Developers and Platform into special features within the VMware Tanzu and Cloud Foundry ecosystem. This Week, Jake Meloche from Aqua Security and field CISO at Tanzu, David Zendzian (aka DMZ) joining the show and deep-dive into all facets of security around Cloud Foundry and Tanzu Application Service.

• VMware Tanzu Application Service (TAS)—New whitepaper!!! Security Outcomes with VMware Tanzu Application Service—This paper explores the many ways TAS helps security teams achieve key outcomes for their apps running on the platform.

If you have concerns about any of your Tanzu environments, or have questions about your enterprise applications and want to understand how we do things within Tanzu or with Tanzu solutions, please reach out and I’d be happy to help. And feel free to share your comments or requests for future discussion topics!