Contributors: Alka Gupta, Ianislav Trendafilov, Tuan Truong, and Andrii Myrgorod
Kubernetes is the leading container orchestration system. Running Kubernetes in production, however, requires the provisioning of infrastructure for compute, networking, and storage as well as components for other day 1 and day 2 operations, such as patching, upgrading, and troubleshooting.
Modern applications are expected to dynamically scale. That expectation requires the underlying infrastructure and the platform to be able to rapidly scale to address the scaling demands of a cloud native application. For true resiliency and fault tolerance, the platform and the underlying infrastructure needs to be continuously monitored with built-in self-healing capabilities.
Early Access
Support for VMware Enterprise PKS in a workload domain is now available for early access in VMware Cloud Foundation 3.8; it not intended to be used in production environments.
VMware Cloud Foundation 3.8 largely automates the deployment of VMware Enterprise PKS through the SDDC Manager. In the latest release, we have integrated cloud management capabilities to deliver a complete private cloud out of the box with automated delivery of composable infrastructure and app services. This solution makes it incredibly easy to deploy and manage VMware Enterprise PKS with built-in lifecycle management of the entire stack. VMware Cloud Foundation automates day 0 to day 2 operations from deployment, configuration, and cloud infrastructure provisioning to upgrading and patching. VMware Enterprise PKS on VMware Cloud Foundation lets you rapidly provide Kubernetes to developers.
After a quick overview of the solution’s integration and architecture, this blog post demonstrates how to deploy VMware Enterprise PKS on VMware Cloud Foundation 3.8. The deployment involves three high-level procedures, which are broken up into three sections in this blog:
- The first procedure describes the prerequisites for deploying VMware Enterprise PKS on VMware Cloud Foundation, including how to contact VMware to participate in the early access program.
- The second procedure demonstrates how to prepare VMware NSX-T Data Center for the deployment.
- The third procedure demonstrates how to deploy VMware Enterprise PKS on VMware Cloud Foundation.
Keep in mind that the exact steps you will follow might be slightly different from this demonstration because of the nature of your environment and any instructions that you might receive as part of participating in the early access program.
Integration and Architecture
VMware Cloud Foundation 3.8 lets you deploy containers-as-a-service infrastructure by using the PKS deployment workflow in SDDC Manager 3.8. This feature deploys VMware Enterprise PKS 1.4.1 in a workload domain on VMware NSX-T Data Center 2.4.1.
In this solution, both VMware NSX and NSX-T are in use. The management domain will continue to use vSphere Distributed Switches and port groups for all the vSphere management networks. NSX-T infrastructure on compute workload domains will provide customers with network virtualization capabilities for VMware Enterprise PKS.
NSX-T is implemented in a shared edge and compute cluster in an SDDC that is compliant with VMware Validated Design for a software-defined data center. This design is basically considered for saving the use of a vSphere cluster specifically for the NSX-T Edge nodes.
Let’s take a closer look at how this solution is constructed. The following diagram provides an overview of the deployment design of VMware Enterprise PKS on VMware Cloud Foundation:
Here’s a diagram that shows the recommended the deployment workflow for bringing up VMware Enterprise PKS on VMware Cloud Foundation:
Prerequisites for Deploying VMware Enterprise PKS on VMware Cloud Foundation
There are several prerequisites for deploying VMware Enterprise PKS on VMware Cloud Foundation:
- Contact VMware to participate in the early access program; see https://kb.vmware.com/s/article/71065.
- Deploy VMware Cloud Foundation in your environment by following the VMware Cloud Foundation deployment guide.
- Download NSX-T and VMware Enterprise PKS bundles using the SDDC Manager and add an NSX-T license key.
- Create a compute workload domain with NSX-T.
The diagram below represents the network topology used for deploying VMware Enterprise PKS on VMware Cloud Foundation. The VMware Enterprise PKS components are installed in a Logical Switch over a Tier-1 router. The pod and node networks are configured over different Tier-1 routers. All Tier-1 routers needs to be connected to the physical network over a Tier-0 router with a proper uplink configuration.
Preparing NSX-T Data Center for deploying VMware Enterprise PKS
1. Deploy and Configure NSX-T edge nodes for the compute workload domain:
- Deploy a minimum of 2 edge nodes with a Large configuration and register them with NSX-T manager.
- Create segments for the TEP network and the uplink network.
- Connect the first interface to the Management segment, the second to the VTEP network, and the third and fourth to the uplink network. (Note: The TEP segment and External uplink segment should be in trunked mode.)
2. Configure an Edge Transport Node:
- Create a Transport zone for the Edge Transport nodes.
- Create two Uplink profiles for the Edge Transport Nodes: One for the overlay network and the other for the external network (north-south communication).
- Create an IP pool for NSX-T edge VMs.
- Configure an Edge Transport Node for both the NSX-T edge VMs: Select the overlay and VLAN transport zone.
- In the next N-VDS tab, select the newly created edge-vtep-ip-pool and overlay uplink profile for overlay network and use the second interface of the edge VM for the TEP network.
- Similarly, select the edge-vlan uplink profile created earlier and link it with the third or fourth interface of the edge VM for the external network.
- Check VTEP to VTEP connectivity between hosts and edge nodes with the jumbo frames by using SSH to connect to the NSX-T-prepared compute ESXi hosts and run the following commands:
esxcfg-vmknic -lRetrieve the IP address of the vmk10 interface and use it in the following command:vmkping ++netstack=vxlan <destination vtep IP> -s 1600
3. Configure NSX-T Logical networking using an Advanced Networking & Security policy:
- Create IP pools for a floating Kubernetes network; the IPs from this IP pool will be assigned to the Kubernetes cluster and used for accessing the Kubernetes cluster, so the subnet used here must be a routable network.
- Create IP blocks for Kubernetes nodes and pods.
- Create and configure a Tier-0 gateway and connect it to the external routable segment.
- Enable BGP routing or add static routes on the Tier-0 gateway.
- Enable route redistribution on the Tier-0 gateway.
- Create and configure a Tier-1 gateway and link it to the Tier-0 gateway.
- Create T1 logical router ports for the PKS control plane VMs and the Kubernetes workload and attach them to the respective segments.
- Enable Route advertisement on the Tier-1 gateway.
Deploying VMware Enterprise PKS on VMware Cloud Foundation
Create two or more resource pools on the compute workload cluster for Availability Zones.
Using any certificate generation tool create certificate chain for NSX-T superuser, Pivotal Operations Manager, VMware Enterprise PKS Management VM, and Harbor using the fully qualified domain name. Now VMware Enterprise PKS should be available in SDDC Manager for the deployment.
Start the deployment of PKS by clicking Get Started in SDDC Manager. Check and validate all the prerequisites.
Fill in the required general settings. Note: The default username ‘ubuntu’ and credentials will be used for authenticating with the PKS API.
Select the NSX-T networking information under the NSX-T settings.
Fill in the configuration details for Pivotal Operations Manager, VMware Enterprise PKS, and Harbor.
Upload the earlier generated certificates (root ca, NSX-T, Pivotal Operations Manager, PKS, and Harbor).
Fill in the networking details for the VMware Enterprise PKS management components.
Create the Kubernetes network.
Create an availability zone for the Kubernetes cluster.
Review all the details.
Validate with no error status and then begin the installation of the PKS components.
After successful completion of the deployment, the details of the VMware Enterprise PKS components can be viewed in the SDDC Manager.
Summary
As seen in this demonstration, VMware Cloud Foundation automates the deployment of NSX-T and VMware Enterprise PKS components through the SDDC Manager. Once deployed, creation of Kubernetes clusters can be automated either by using the VMware vRealize Automation catalog service or through other CI/CD workflows leveraging the PKS CLI.
Support for VMware Enterprise PKS 1.4.1 is offered as an early access feature in VMware Cloud Foundation 3.8 and is not intended to be used in production environments.
VMware Validated Design for VMware Enterprise PKS is generally available. It can be used as a possible container service architecture for VMware Enterprise PKS on VMware Cloud Foundation with NSX-T workload domains.
About the Author
More Content by Prasad Kalpurekkal
