HAProxy is an optional load balancer included in the canonical open source Cloud Foundry deployment. Its intended use is on IaaSes (Infrastructures as a Service) that do not offer built-in load balancers . On vSphere, this means without the optional network virtualization solutions, NSX-T and NSX-V. This blog post describes how to assign an IPv6 address to an HAProxy load balancer in a Cloud Foundry deployment.
Users following this blog post should be familiar with BOSH, BOSH’s manifest operations files, IPv6, and deploying Cloud Foundry using cf-deployment.
2. Set Up DNS Records
We set up our DNS (Domain Name System) records as shown in the table below
(we’re using the same domain,
cf.nono.io, for both system and app domains, and
this is probably not a good
|Hostname||IPv4 Address||IPv6 Address|
Our DNS server uses a BIND-format (Berkeley Internet Name Domain)  file, and here are the raw entries (tweaked for readability):
cf.nono.io. A 10.0.250.10 AAAA 2601:646:100:69f5::10 *.cf.nono.io. A 10.0.250.10 AAAA 2601:646:100:69f5::10
Note that the IPv4 address (10.0.250.10) is in a private
network and not reachable from
the internet, but the IPv6 address (2601:646:100:69f5::10) is in a public one.
Indeed, the IPv6 address is in a
/64 subnet, one of 8 subnets that Comcast has
Only IPv6-enabled clients can reach our Cloud Foundry; IPv4-only clients can’t.
3. Prepare the Cloud Foundry Deployment
Let’s download what we need and log in to our BOSH Director:
mkdir -p ~/workspace cd ~/workspace git clone https://github.com/cloudfoundry/cf-deployment git clone https://github.com/cunnie/deployments git clone https://github.com/cloudfoundry/cf-acceptance-tests.git export BOSH_ENVIRONMENT=vsphere # our vSphere BOSH Director's environment alias bosh login cd deployments
- the GitHub repo
cf-deploymentcontains the BOSH manifest and manifest operations files to deploy Cloud Foundry.
- the GitHub repo
deploymentscontains additional scripts and manifest operations files to deploy Cloud Foundry with an IPv6 HAProxy.
- the GitHub repo
cf-acceptance-testscontains a sample CF app that we can push to test IPv6 connectivity.
3.0 Prepare the BOSH Cloud Config
Normally we’d add our IPv6 Network to our Cloud Config’s YAML file and then type
bosh update-cloud-config, but that’s not what we’re going to do in this
case. Instead, we’re going to use the Cloud Config included cf-deployment and
use a custom manifest operations files to add our IPv6 network to it.
Here are the relevant lines from our script:
# We don't use the primary cloud config (we already have one); instead, # we set up a secondary config bosh update-config \ --non-interactive \ --type cloud \ --name cf \ <(bosh int \ -o $DEPLOYMENTS_DIR/cf/cloud-config-operations.yml \ -l $DEPLOYMENTS_DIR/cf/cloud-config-vars.yml \ $DEPLOYMENTS_DIR/../cf-deployment/iaas-support/vsphere/cloud-config.yml)
The environment variable
$DEPLOYMENTS_DIR refers to the directory which
contains our customizations (
Let’s talk about our customizations; We customize as follows:
We set up our variables in
cf/cloud-config-vars.yml. The file is uninteresting—it sets up the IPv4 subnets for the three AZs, two of which (
az3) will be immediately removed as part of our manifest operations.
We apply our manifest operations (
We remove two of the AZs, leaving one,
az1. We don’t want a sprawling deployment; Our vSphere cluster is too small. We can only fit one AZ.
az1’s' subnet to include a static IP address, 10.0.250.10, which will be assigned to the HAproxy, and which is the DNS
We introduce the IPv6 subnet, 2601:646:100:69f5::/64, including a static IPv6 address, 2601:646:100:69f5::10, which will be assigned to the HAproxy, and which is the DNS
3.1 Prepare the BOSH Runtime Config
As long as BOSH DNS is colocated on all deployed VMs (which is the default BOSH Runtime Config), you shouldn’t need to make changes.
3.2 Deploy Cloud Foundry with BOSH
Deploying the VMs requires one command,
bosh deploy, but with many options.
Here are the relevant lines from our deploy shell
bosh \ -e vsphere \ -d cf \ deploy \ --no-redact \ $DEPLOYMENTS_DIR/../cf-deployment/cf-deployment.yml \ -l <(lpass show --note cf.yml) \ -v system_domain=cf.nono.io \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/scale-to-one-az.yml \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-haproxy.yml \ -o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-latest-stemcell.yml \ -o $DEPLOYMENTS_DIR/cf/letsencrypt.yml \ -o $DEPLOYMENTS_DIR/cf/haproxy-on-ipv6.yml \ -v haproxy_private_ip=10.0.250.10 \ --var-file=star_cf_nono_io_crt=$HOME/.acme.sh/\*.cf.nono.io/fullchain.cer \ --var-file=star_cf_nono_io_key=$HOME/.acme.sh/\*.cf.nono.io/\*.cf.nono.io.key \
-o $DEPLOYMENTS_DIR/cf/haproxy-on-ipv6.ymlis the most important line for deploying HAProxy with IPv6. It’s a BOSH manifest operations file, and will be covered in the following section.
-v haproxy_private_ip=10.0.250.10sets the HAProxy to the IPv4 address pointed to by
-v system_domain=cf.nono.iosets the system domain, which should point to the IPv4 address above.
<(lpass show --note cf.yml)is a YAML file that sets only one property,
cf_admin_password. The author prefers to have a easy-to-remember admin password rather than needing to extract the password from a BOSH-generated CredHub secret. This line, this YAML file, is unnecessary; you may safely skip it.
Any parameter beginning with
$DEPLOYMENTS_DIR/../cf-deploymentrefers to a file in cf-deployment GitHub repo, which means it’s a canonical manifest or manifest operations file.
-o $DEPLOYMENTS_DIR/../cf-deployment/operations/use-haproxy.yml. Enable HAProxy, otherwise there’s no VM to which we can assign an IPv6 address.
-o $DEPLOYMENTS_DIR/cf/letsencrypt.ymlis a custom manifest operations file the deploys HAProxy with a valid, commercial TLS certificate issued by Let’s Encrypt. Its usage outside the scope of this document, but if enough are interested the author may write a blog post to describe how to deploy Cloud Foundry with a free Let’s Encrypt wildcard certificate. You may safely skip this file.
--var-file=star_cf_nono_io_keyare also related to the Let’s Encrypt TLS certificate. You may skip them.
4. Prepare the Manifest Operations File to Enable IPv6 on HAProxy
The 23-line manifest operations file is too long to inline in this blog post, but let’s review key points:
- Assign HAProxy VM an IPv6 address. BOSH requires IPv6 to have leading zeros,
no double-colons. In other words, don’t abbreviate the IPv6 address. This
IPv6 address should be the AAAA DNS record of the app & system domains (e.g.
# haproxy has an IPv6 address - type: replace path: /instance_groups/name=haproxy/networks/name=PAS-IPv6? value: name: PAS-IPv6 static_ips: - 2601:0646:0100:69f5:0000:0000:0000:0010
- Configure HAProxy job (process) to bind to both IPv4 and IPv6 addresses:
# configure haproxy to bind to the IPv6 in6addr_any address "::" - type: replace path: /instance_groups/name=haproxy/jobs/name=haproxy/properties/ha_proxy/binding_ip? value: "::" # configure haproxy to bind to both IPv4 & IPv6 interfaces - type: replace path: /instance_groups/name=haproxy/jobs/name=haproxy/properties/ha_proxy/v4v6? value: true
We push an application, then
curl the application over IPv6.
cf api api.cf.nono.io --skip-ssl-validation # replace your CF API endpoint here cf login cf target -o system -s system # or whichever org & space you want to use cf push dora -p ../cf-acceptance-tests/assets/dora curl -6 -k https://dora.cf.nono.io # replace your application domain here # "Hi, I'm Dora!"
A reasonable question is, “do browsers have trouble consistently navigating to
HAProxy’s IPv6 address given that
*.cf.nono.io resolve to both IPv4 and IPv6
We haven’t experienced trouble with either the Google Chrome or Firefox browsers. Similarly, we haven’t experienced problems with the Cloud Foundry CLI (Command Line Interface); however, we suspect that if we had a machine on our internal network whose IPv4 addressed matched the IPv4 address of our HAProxy (i.e. 10.0.250.10), then connectivity would be erratic.
Assigning the HAProxy VM’s default gateway to the IPv4 or IPv6 interface is a nuanced decision. If the BOSH Director is in a different IPv4 subnet, you must assign the default gateway to the IPv4 interface so that the VM can reach the BOSH director and retrieve its configuration and releases (as is the case in our setup).
If you have IPv6 router advertisements on your IPv6 subnet, your IPv6 interface will pick up its default gateway for free. If you do not have router advertisements, you will need a more complex manifest operations file; check here for inspiration.
For best results, use Ubuntu Xenial stemcells 621.51 or greater; IPv6 router advertisement are enabled by default.
Most IaaSes (Amazon Web Services (AWS), Google Cloud, and Microsoft Azure) offer load balancers, and they typically charge $200+/yr for the service.
The author is of the opinion is that load balancers are often an unnecessary expense, and many users are better off with a carefully-monitored single webserver rather than a fleet of webservers fronted by a load balancer. To quote Andrew Carnegie:
put all your eggs in one basket, and then watch that basket
BIND is an old-school DNS server. Nowadays most users are better off using DNS-as-a-service (e.g. Amazon’s Route 53, Google Cloud DNS), but for the hardcore, deploying your own DNS servers is a wonderfully addictive option. The author, for example, has been running his own DNS servers since 1996, and currently maintains 4 DNS servers on 4 different IaaSes (Azure, AWS, Google, Hetzner) on 3 different continents (America, Asia, Europe).
The author is an unapologetic DNS fanboy, and had Paul Mockapetris, the author of the original DNS RFC, autograph his laptop with a permanent marker.