VMware Enterprise PKS Automates Operations at Scale for Node Failures

July 9, 2019 Joseph Griffiths

One of the most powerful features of VMware Enterprise PKS is its capability to manage desired state for Kubernetes clusters. This capability is provided in part by BOSH. For example, consider the following three-worker node cluster deployed in VMware Enterprise PKS:

NAME                                   STATUS   ROLES    AGE    VERSION
0c04b65b-1215-49a6-b4f4-383d75ff958a   Ready       86d    v1.12.4
c14736b9-2b54-484c-b783-a79453e28804   Ready       134m   v1.12.4
fb1cd9ad-568e-4e83-a7a8-738dfe050712   Ready       84d    v1.12.4

On this Kubernetes cluster, I have deployed a simple web application with four pods and an external load balancer:

kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
bootcamp-95bd888fc-8j77b   1/1     Running   0          40s
bootcamp-95bd888fc-dbfb7   1/1     Running   0          63s
bootcamp-95bd888fc-dzkmc   1/1     Running   0          40s
bootcamp-95bd888fc-tmwfz   1/1     Running   0          40s

kubectl get deployments
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
bootcamp   4         4         4            4           87s

kubectl get services
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP               PORT(S)          AGE
bootcamp     LoadBalancer   10.100.200.223   10.40.14.41,100.64.96.7   8080:32013/TCP   2m7s

What is the impact of a worker node failure?

A worker node is marked as Condition Unknown after the master is unable to reach the node's kubelet agent. If the node continues in Condition Unknown, it will evict the pods. The eviction process will restart pods on other nodes if possible. At this point, Kubernetes has restored the container applications to service but we are left with a Condition Unknown worker node.

The desired state of the Kubernetes cluster is to have three worker nodes. VMware Enterprise PKS restores the desired state for the Kubernetes cluster. In order to simulate this function of VMware Enterprise PKS, we will power off a worker node.  

Simulating a node failure

We can identify worker nodes in VMware vSphere by reviewing the custom attributes and looking for worker in the job field:  

Powering off this worker node will produce a warning from vSphere that it is managed by BOSH:  

Ignoring the well-placed warning, the machine is powered off. Soon after being powered off, the worker node is marked as NotReady or Condition Unknown in Kubernetes:

kubectl get nodes
NAME                                   STATUS     ROLES    AGE    VERSION
0c04b65b-1215-49a6-b4f4-383d75ff958a   Ready         86d    v1.12.4
c14736b9-2b54-484c-b783-a79453e28804   Ready         141m   v1.12.4
fb1cd9ad-568e-4e83-a7a8-738dfe050712   NotReady      84d    v1.12.4

After a reasonable time, Kubernetes rebuilds the failed pods on another node (as shown by the difference in age):

kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
bootcamp-95bd888fc-dbfb7   1/1     Running   0          13m
bootcamp-95bd888fc-jfg82   1/1     Running   0          4m51s
bootcamp-95bd888fc-s797b   1/1     Running   0          4m51s
bootcamp-95bd888fc-tmwfz   1/1     Running   0          13m

VMware Enterprise PKS enforces the desired state of the cluster by replacing the powered off node with a newly deployed node and removing the failed node. In a traditional Kubernetes environment, the replacement of a failed node is a manual process. VMware Enterprise PKS excels at providing high scalability in part through its high degree of automated day-two operations.

About the Author

Joseph Griffiths is a virtualization focused architect who has deployed and architected complex cloud based solutions. Joseph is business driven with incredible people skills, with a determination to provide well documented repeatable results. Joseph is honest and hardworking with an interest in all technology. Joseph has managed many technical projects and implementations to successful completion. Joseph also has the honor of being a double VMware certified design expert (#143). Joseph enjoy's public speaking and education opportunities. He blogs about technology at blog.jgriffiths.org and can be found on Twitter @Gortees.

More Content by Joseph Griffiths
Previous
PAS on Kubernetes Improves the Developer Experience
PAS on Kubernetes Improves the Developer Experience

PAS will run on Kubernetes, and Kubernetes already is best delivered by VMware PKS. So, any enterprise that...

Next
Using the Crunchy PostgreSQL Operator to Deploy a PostgreSQL Cluster on VMware Enterprise PKS
Using the Crunchy PostgreSQL Operator to Deploy a PostgreSQL Cluster on VMware Enterprise PKS

This blog showcases how you can deploy an enterprise-grade PostgreSQL cluster on-demand using the Crunchy P...

How To Think Cloud Native

Learn more