Exploring kube-apiserver load balancers for on-premises Kubernetes clusters

May 22, 2019 Chad Moon

When you bootstrap a Kubernetes cluster in a non-cloud environment, one of the first hurdles to overcome is how to provision the kube-apiserver load balancer. If you are running a non-HA single node control plane cluster, the load balancer is unnecessary because all API requests are directly routed to the single control plane. In highly available configurations, a load balancer must sit in front of the kube-apiservers to correctly route requests to healthy API servers.

With cloud platforms such as AWS, it’s trivial to click a few buttons and launch an elastic load balancer. Outside of these platforms, the solution is less obvious. There are, however, load balancing options that can be deployed in non-cloud environments.

First, let’s review why the kube-apiserver load balancer is necessary.  

As seen above, the load balancer routes traffic to the kube-apiservers. If a kube-apiserver goes down, the load balancer routes traffic around this failure.

Worker nodes communicate with the control plane through a single API endpoint. Using a load balancer for the endpoint ensures that API requests are properly distributed to a healthy kube-apiserver. If there were no load balancer in place, each worker would need to choose a specific kube-apiserver to communicate with. If this kube-apiserver were to fail, it would cause cascading failures to the bound worker nodes, which is the opposite of high availability.

In the rest of this blog post, we’ll discuss several options for implementing a kube-apiserver load balancer for an on-premises cluster, including an option for those running Kubernetes on VMware vSphere.

DNS for Load Balancing

A common scenario is to use round-robin DNS as a load balancer. This method carries several disadvantages. The lack of health checks prevents routing around failed servers. Unpredictable caching in the DNS hierarchy, as well as client-side, make management and updates difficult. Because of these drawbacks, there are better options to explore.

Option One: Standalone HAProxy

HAProxy is a quick and easy option to use. After installing the package on your load balancing server, you configure the list of kube-apiservers along with their health checks. Here’s an example configuration that balances three kube-apiservers, 10.10.10.10, 10.10.10.11 and 10.10.10.12.

global
   log /dev/log	local0
   log /dev/log	local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
   stats timeout 30s
   user haproxy
   group haproxy
   daemon

   # Default SSL material locations
   ca-base /etc/ssl/certs
   crt-base /etc/ssl/private

   # Default ciphers to use on SSL-enabled listening sockets.
   # For more information, see ciphers(1SSL). This list is from:
   #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
   # An alternative list with additional directives can be obtained from
   #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
   ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
   ssl-default-bind-options no-sslv3

defaults
   log	global
   mode	http
   option	httplog
   option	dontlognull
       timeout connect 5000
       timeout client  50000
       timeout server  50000
   errorfile 400 /etc/haproxy/errors/400.http
   errorfile 403 /etc/haproxy/errors/403.http
   errorfile 408 /etc/haproxy/errors/408.http
   errorfile 500 /etc/haproxy/errors/500.http
   errorfile 502 /etc/haproxy/errors/502.http
   errorfile 503 /etc/haproxy/errors/503.http
   errorfile 504 /etc/haproxy/errors/504.http

frontend k8s-api
   bind 0.0.0.0:6443
   mode tcp
   option tcplog
   default_backend k8s-api

backend k8s-api
   mode tcp
   option tcplog
   option tcp-check
   balance roundrobin
   default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

       server apiserver1 10.10.10.10:6443 check
       server apiserver2 10.10.10.11:6443 check
       server apiserver3 10.10.10.12:6443 check

A drawback of this option is that it makes the instance running HAProxy a single point of failure. The whole point of high availability is to remove the single point of failure, not introduce one. Other than quick lab clusters or clusters for proof of concepts, this approach is not recommended.

Option Two: The Keepalived Package with HAProxy

Keepalived is a powerful package that leverages the Linux kernel feature of floating IP addresses through Virtual Router Redundancy Protocol (VRRP). Two instances of HAProxy are launched, a primary instance and a standby instance. If the primary instance fails, Keepalived moves, or “floats,” the IP address to the standby, and no service disruption will occur.

Another interesting way to leverage VRRP is to run it inside a Docker container. These containers can be run as static pods paired with HAProxy, or even directly on the control plane nodes themselves. By using static pods, you can benefit from maintaining your load balancing solution with Kubernetes manifests, just like you do with your workloads.

The drawbacks with Keepalived are that you still need to choose and maintain the actual load balancing software. Keepalived provides only the floating IP address functionality.

Option Three: vSphere HA

Leveraging the powerful high availability features of VMware vSphere High Availability is yet another option. The HA setting can be enabled on the cluster level as shown below. 

If a physical ESXi host fails, vSphere HA automatically restarts the HAProxy VM on a healthy ESXi host within the cluster, while maintaining the same IP address.

A great thing about the vSphere HA feature is that it works on the VM level so it’s agnostic to what software inside. There is also no need for specialized configuration because vSphere handles it for us.

Note that you must use a shared datastore for VMs to successfully float between hosts. VMware vSAN is a great choice here, but external options such as iSCSI will also work.

A downside is the speed of the operation. If the HAProxy VM is running on a host that experiences an immediate failure, there will be a delay due to the time needed for the failure detection and VM boot time on a healthy host. The HAProxy endpoint will be non-responsive until the operation completes.

It’s possible to prevent the downtime period by enabling vSphere Fault Tolerance (FT) on the HAProxy VM. In this case, a secondary “shadow” VM runs on a separate ESXi host. The VM is constantly replicated from the primary VM over the network. If the primary VM or its host fails, the IP address will instantly float to the secondary VM. In this scenario, no downtime will be observed during the failover process.

Bonus Option: VMware NSX-T

NSX-T is an extremely powerful, fully featured network virtualization platform that includes built-in load balancing. VMware Enterprise PKS leverages it out of the box, but it’s possible to install and configure it for VMware Essential PKS as well. An added benefit of using NSX-T load balancers is the ability to be deployed in server pools that distribute requests among multiple ESXi hosts. In this scenario, there would be no downtime if an individual host failed.

Conclusion

As shown above, there are multiple load balancing options for deploying a Kubernetes cluster on premises. Here’s a quick summary of each option’s main advantage:

  • For a quick POC, the simplicity of HAProxy can’t be beat.
  • For a highly available setup on bare metal, using HAProxy with Keepalived is a reliable option.
  • On vSphere, you can take advantage of vSphere HA and combine it with a shared datastore to run your load balancing VMs.
  • If NSX-T is available in your cluster, load balancers can easily be created with a click of a button.

About the Author

Chad Moon is a Customer Reliability Engineer for Essential PKS. He specializes in advising, assisting, and guiding enterprise companies on their Kubernetes journey via proactive and reactive engineering.

More Content by Chad Moon
Previous
Simplifying TLS for Kubernetes Services
Simplifying TLS for Kubernetes Services

In this blog series, I’ll highlight an easy path forward for operations teams that need to up their certifi...

Next
VMware Announces PKS Partner Application Program
VMware Announces PKS Partner Application Program

The VMware PKS Partner Application Program helps application software vendors that fully support their prod...

SpringOne. Catch all the highlights

Watch now