Windows Containers in Cloud Foundry? Here's How We Did It

Hey, have you heard? Pivotal Cloud Foundry now supports running applications in Windows Server Containers.

The platform has supported .NET apps on Windows Server 2012 R2 for two years. This move is a significant step for .NET developers, continuing to improve on the .NET developer experience.

That effort got a huge boost with Windows Server 2016 and its native support for Windows Server Containers. “Native” is the key term; it enables a far more robust .NET developer experience in Pivotal Cloud Foundry. Specifically, with native Windows Server Containers developers can:

• Use more platform features (compared to the earlier version based on 2012 R2)
• Enjoy improved isolation and resource management for their apps

We launched this effort months ago, and planned to support Windows Containers from Windows Server 1709. And that’s just what we’ve done!

Garden-runC Release and Windows

Previously, for applications running on the windows2012R2 stack, the garden-windows team implemented a Windows backend to the garden API and maintained garden-windows-bosh-release. This allowed the team to have flexibility and ownership of their own code base. But, we felt some pains when integrating with new platform features.

We wanted to improve our development process when building our Garden implementation for Windows Server 2016. So, we prioritized more cross-team collaboration with the core Garden team. We also wanted to adopt the OCI standards for container lifecycle and container image management. (We build to this standard for Linux on Cloud Foundry, so it makes sense to do this for Windows too).

To this end, we chose to integrate with Garden-runC Release and Guardian rather than maintain another BOSH release.

The pluggable nature of Guardian means we can focus on solely implementing the Runtime, Network, and Image plugins for Windows Server 2016:

• winc: the OCI Runtime Plugin for creating and running containers in windows2016 stack.
• winc-network: the Network Plugin for Guardian.
• groot-windows: the Image Plugin for managing container images and setting up a root filesystem. This plugin can be used with local OCI image or remote docker URIs.

Here’s a quick diagram, showing how these components interact.

You’ll notice that the Garden components on a Windows Server 2016 Diego cell are the same as those on a Linux Diego cell…except for the new plugins mentioned above.

Windows Container Isolation

Windows Server 2016 comes with Windows Server Containers. The primitives here are similar to those Linux kernel. We use the official hcsshim Go library for instrumenting the Windows Server Containers in Cloud Foundry. The underlying container operations are performed by the Windows Host Compute Service (HCS). Windows Server Containers provide application isolation through process and namespace isolation technology. A Windows Server container shares a kernel with the container host and all containers running on the host.

What about security? Here’s how Garden-Windows secures applications:

• Filesystem isolation: By default, Windows Server Containers use a temporary scratch space on the system drive of the container host. This space is used for storage during the life of the container. This allows us to have a dedicated filesystem per container (i.e. each container has its own C:\ drive). Any modifications made to the container filesystem or registry are captured in a sandbox that’s managed by the host. As a result, we are now able to create and maintain a root filesystem (rootfs) that allows applications to receive security upgrades when running on the platform.
• Disk usage: Disk usage limits are enforced by NTFS disk quotas in both windows2012R2 and windows2016 stack. Previously, this quota applied to all files owned by a given user. But, now with Windows Server Containers the quota is applied to the sandbox image. (We used disk quotas because HCS currently doesn’t have support for enforcing disk limits.)
• Network isolation: Each Windows Server Container has a network compartment that’s analogous to Linux containers. Containers function similarly to virtual machines in regards to networking. Each Container has a virtual network adapter (vNIC) connected to a virtual switch.
• Memory usage: HCS provides an API to set memory limits on containers at creation time.
• CPU usage: Configuring CPU shares allocated to containers is now possible in Windows Server Containers via HCS.

Future Work

There’s a .NET Renaissance underway, and we’re excited to be a part of it. That’s why we’re continuing to contribute to the cloud-native .NET movement, including .NET Core and Steeltoe.

The best part? Native Windows Server Containers feature makes it easier for us to add new capabilities. In the coming months, we plan to add support for Container-to-Container networking and Volume Services for Windows Server Containers

We hope this post serves as a good introduction to containers as we’ve implemented on Windows Server 2016. What are your plans for .NET and Cloud Foundry? We want to hear from you! Visit us on Slack and tell us about your scenarios.