0. Abstract
BOSH is a VM orchestrator; a BOSH Director creates, configures, monitors, and deletes VMs. The BOSH Director interoperates with a number of IaaSes (Infrastructure as a Service), one of which is VMware vSphere, a virtualization platform. BOSH traditionally operates exclusively within the IPv4 networking space (i.e. the BOSH Director has an IPv4 address (e.g. 10.0.0.6), and the VMs which it deploys also have IPv4 addresses); however, recent changes have enabled IPv6 within the BOSH Framework.
In this blog post we show how we deployed a BOSH Director with an IPv6 address (no IPv4), and, in turn, used the BOSH Director to deploy several VMs with IPv6 addresses.
We expect this blog post to be of interest to those who plan to deploy BOSH in IPv6-enabled environments.
1. Prerequisites
Use at least the following versions:
- Stemcell 3468.13 Ubuntu/Trusty [Ubuntu]
- BOSH Director 264.5.0
- BOSH CLI 2.0.45
bosh-deployment
commit 218e6d50
The following must have IPv6 addresses [Hybrid]
- the workstation from which the BOSH Director is redeployed
- the VMware vCenter
- the VMware ESXi host
2. Deployment Overview
[Ubuntu] We haven’t yet made changes to the
bosh-agent
to accommodate IPv6 on the CentOS-flavored stemcells; pull requests
are welcome.
History
Enabling IPv6 on BOSH was a side project we started a year ago, grossly underestimating the amount of time required — we thought it would take a couple of weeks at most; it took over a year. The changes spanned several BOSH components: the BOSH Director (e.g commit 4a35c4b8), the BOSH agent (e.g. commit 0962dce7), the BOSH CLI (e.g. commit 0316b3a5), and BOSH deployment (e.g. commit 214ebac4).
Although both our names appeared on the commits, Dmitriy did much of the heavy lifting. The code is his. Brian contributed the IPv6-enabled vSphere infrastructure and network & kernel configuration requirements.
Gotchas
Don’t abbreviate
IPv6 addresses in BOSH manifests or Cloud Configs. Don’t use double ::
, don’t
strip leading zeroes. As an extreme example, the loopback address (::1
) should be
represented as 0000:0000:0000:0000:0000:0000:0000:0001
.
Don’t use large reserved
IP
ranges (> 1k IP addresses); they will cause bosh deploy
to hang. We have
cheerfully named the following Cloud Config “the fifth horseman of the BOSH
apocalypse”, for no deployment to that network will ever complete.
networks:
- name: IPv6
type: manual
subnets:
- range: 2601:0646:0100:69f1:0000:0000:0000:0000/64
gateway: 2601:0646:0100:69f1:020d:b9ff:fe48:9249
dns:
- 2001:4860:4860:0000:0000:0000:0000:8888
- 2001:4860:4860:0000:0000:0000:0000:8844
# This large range will cause `bosh deploy` to hang; don't do it
reserved: [ 2601:0646:0100:69f1:0000:0000:0000:0000-2601:0646:0100:69f1:ffff:ffff:ffff:ffff ]
Don’t upload stemcells using the bosh.io URLs (bosh upload-stemcell https://s3.amazonaws.com/bosh-core-stemcells/vsphere/bosh-stemcell-...
); they
won’t work (Network is unreachable
); Amazon S3 doesn’t have IPv6 addresses for
s3.amazonaws.com. Instead, download the stemcell locally to your workstation,
then upload the stemcell from your workstation to your Director (e.g. bosh upload-stemcell stemcell.tgz
).
BOSH requires the IPv6 default route to reside in the same subnet as the gateway
(which is not an IPv6 requirement (often the default route is an fe80::...
address), though it is an IPv4 requirement).
IPv6’s Neighbor Discovery Protocol may subvert the BOSH networking model. For example, on a multi-homed VM with both IPv4 interface and IPv6 interfaces, with IPv4 interface being set as the default gateway via BOSH, and a gateway assigned to the IPv6 via Router Advertisement, may result in non-local traffic going out both the IPv4 and IPv6 interfaces instead of solely the IPv4 interface. Some may view this as a feature.
Certain versions of the vCenter Appliance require modifying
/etc/sysctl.conf
to enable IPv6.
BOSH doesn’t have a concept of “dual-stack”. In other words, when it deploys a VM, the VM’s network interface can have one IP address, either IPv4 or IPv6 but not both.
BOSH won’t allocate certain addresses, e.g. “subnet zero”.
Footnotes
[Ubuntu] IPv6 only works on Ubuntu stemcells; we
haven’t yet made changes to the bosh-agent
to accommodate IPv6 on the
CentOS-flavored stemcells. Pull requests are welcome.
[Hybrid] The BOSH director and its VMs can be deployed in a hybrid manner, with both IPv4 and IPv6 addresses. Such deployments do not always require the workstation, vCenter, and ESXi host to have IPv6 addresses. Discussions of such deployments are outside the scope of this blog post.