BOSH is a VM orchestrator; a BOSH Director creates, configures, monitors, and deletes VMs. The BOSH Director interoperates with a number of IaaSes (Infrastructure as a Service), one of which is VMware vSphere, a virtualization platform. BOSH traditionally operates exclusively within the IPv4 networking space (i.e. the BOSH Director has an IPv4 address (e.g. 10.0.0.6), and the VMs which it deploys also have IPv4 addresses); however, recent changes have enabled IPv6 networking within the BOSH Framework.
In this blog post we show how we deployed a BOSH Director with an IPv4 address (no IPv6), and, in turn, used the BOSH Director to deploy a VM with both IPv4 and IPv6 addresses and which runs an nginx web server. Future blog posts will describe installing a BOSH Director in a pure IPv6 network.
We expect this blog post to be of interest to those who plan to deploy BOSH in IPv6-enabled environments on vSphere.
BOSH with IPv6 is in beta! We urge caution when deploying BOSH with IPv6 – limiting your deployments to test environments is a good idea. We welcome feedback.
Use at least the following versions:
- Stemcell 3468.13 Ubuntu/Trusty [Ubuntu]
- BOSH Director 264.5.0
- BOSH CLI 2.0.45
2. Deployment Overview
3. Deploying the BOSH Director
We set our BOSH Director’s alias to “ipv4” and log in:
# set the alias for our BOSH Director to "ipv4" bosh -e 10.0.9.151 alias-env ipv4 # use something along these lines to find the admin password: # `bosh int --path /admin_password creds.yml` # log in bosh -e ipv4 log-in Email (): admin Password ():
4. Upload the Cloud Config
Assuming that you already have a Cloud Config with an IPv4 network, let’s add an additional Cloud Config that defines the IPv6 network.
bosh -e ipv4 update-config cloud cloud-config-vsphere-ipv6.yml --name ipv6
The IPv6 Cloud Config is shown below, and can also be seen on GitHub.
networks: - name: ipv6 type: manual subnets: - range: "2601:0646:0100:69f1:0000:0000:0000:0000/64" gateway: "2601:0646:0100:69f1:020d:b9ff:fe48:9249" dns: - 2001:4860:4860:0000:0000:0000:0000:8888 - 2001:4860:4860:0000:0000:0000:0000:8844 azs: [z1] cloud_properties: name: IPv6
IPv6 addresses in BOSH manifests or Cloud Configs [why no abbreviations?] .
Don’t use double colons (
::), don’t strip leading zeroes. As an
extreme example, the loopback address (
::1) should be represented
4. Upload the Stemcell and the nginx Release
bosh -e ipv4 us https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-trusty-go_agent?v=3468.17 \ --sha1 1691f18b9141ac59aec893a1e8437a7d68a88038 bosh -e ipv4 ur https://bosh.io/d/github.com/cloudfoundry-community/nginx-release?v=1.12.2 \ --sha1 70a21f53d1f89d25847280d5c4fad25293cb0af9
5. Deploy the Web server
We create a manifest for our deployment; it can be viewed on Github.
We assign our instance group to have two networks within our manifest as follows:
instance_groups: - name: nginx networks: - name: default - name: ipv6 default: [dns, gateway]
Note that we assign our default gateway to the IPv6 interface. This has the implication that our IPv4 interface can only communicate on its local subnet (i.e. 10.0.9.0/24), which means that it must be deployed on the same subnet as the BOSH Director (otherwise the VM would be unable to communicate with the Director, hence would be unable to receive its configuration). [Routing]
Deployment is straightforward:
bosh -e ipv4 -d nginx deploy nginx-ipv46.yml
6. Check It’s Working
We check to make sure our nginx VM is running and that it has configured its IP addresses properly:
bosh -e ipv4 -d nginx instances Using environment 'bosh-vsphere-ipv4.nono.io' as user 'admin' (openid, bosh.admin) ... Instance Process State AZ IPs nginx/821894df-9441-4325-92aa-2f4ded0e2bd9 running z1 10.0.9.165 2601:0646:0100:69f1:0000:0000:0000:0165
We have seen how, using a standard BOSH director with a standard stemcell, we were able to deploy an IPv6-enabled VM running a service (nginx) that was reachable from the internet.
Don’t abbreviate IPv6 addresses in BOSH manifests or Cloud Configs.
Don’t use large
ranges (> 1k IP addresses); they will cause
bosh deploy to hang.
Make sure your application binds to the IPv6 address of your VM if you plan on using the IPv6 endpoint (e.g. http://[2601:646\:100:69f1::165]/). You may need to make additional configuration changes, possibly code changes.
Tech notes: the underlying system call (kernel interface) to create a socket,
socket(2), requires the
specification of the address family, which can either be IPv4 (
AF_INET6), which means that applications need to “opt-in” to binding to
the IPv6 address (it’s not automatic). Certain applications are coded to bind to
both IPv4 and IPv6 addresses seamlessly (e.g.
sshd); however, that’s not the
case for the majority of applications. Even nginx, a popular webserver, requires
a fairly cryptic directive to bind to both IPv4 & IPv6:
listen [::]:80 ipv6only=off; (the directive to listen to IPv4 is a simple
Be aware of the security implications of IPv6 Router Advertisements BOSH stemcells are currently configured to accept IPv6 router advertisements which expose the VM to man-in-the-middle attacks. [Router Advertisements]
IPv6 is enabled on all the VM’s interfaces. Once BOSH
assigns an IPv6 address to an interface on a VM, the other interfaces may pick up
an IPv6 address as well, one that was not assigned by BOSH but rather acquired
via IPv6’s Neighbor Discovery Protocol’s (ND’s) stateless address
(SLAAC). For example, the web server we deployed acquired an additional IPv6
addresses on its “IPv4” interface:
Currently BOSH doesn’t have a concept of “dual-stack”. In other words, when it deploys a VM, BOSH assigns the VM’s network interface either an IPv4 or an IPv6 address, but not both (though an IPv4 interface may acquire an IPv6 address via SLAAC).
Currently BOSH requires the IPv6 default route to reside in the same subnet
as the gateway (often the IPv6 default route is an
BOSH won’t allocate certain addresses, e.g. “subnet zero”.
We began work in January 2017. Each week we picked one day to work in the late evening for three hours. The changes spanned several BOSH components: the BOSH Director (e.g commit 4a35c4b8), the BOSH agent (e.g. commit 0962dce7), the BOSH CLI (e.g. commit 0316b3a5), and BOSH deployment (e.g. commit 214ebac4).
We’d like to thank the many people who made IPv6-on-BOSH possible: the BOSH Development Team (Danny Berger, Chris De Oliveira, Tom Viehman, Eve Quintana, Difan Zhao, Joshua Aresty) for merging the pull requests and fleshing-out the testing structure, Toolsmiths (Mark Stokan, Ken Lakin, and Der Wei Chan) for creating the necessary environments, and IOPS (Sachin Prasad, Quintin Donnelly, and Pablo Lopez) for enabling IPv6.
[Ubuntu] IPv6 only works on Ubuntu Trusty stemcells; we
haven’t yet made changes to the
bosh-agent to accommodate IPv6 on the
CentOS-flavored or Ubuntu Xenial stemcells. Pull requests are welcome.
[why not dual stack?] Our deployed webserver VM is multihomed — it has two network interfaces: one which has the IPv4 address (10.0.9.165), and the other which has the IPv6 address (2601:646:100:69f1::165). But a more common approach is to use a single, dual stack, network interface:
A dual-stack device is a device with network interfaces that can originate and understand both IPv4 and IPv6 packets.
So why did we opt for the dual-homed single-stack approach instead of the single-homed, dual stack approach? The answer is that BOSH’s networking model assumes one and only one IP address (be it IPv4 or IPv6) is assigned to a given network interface. To accommodate dual stack we would have had to make changes to BOSH vSphere CPI and BOSH Agent - changes that would have required time we did not have. The multihomed single-stack approach was an expedient and technically valid choice.
[Routing] Most non-BOSH-deployed machines with
both IPv4 and IPv6 addresses have two default routes: one for IPv4, and one for
IPv6 (we discount the
link-local addresses (i.e.
fe80::/10) which, by definition, don’t have a route). The BOSH networking
model, as it currently stands, only allows one default route.
This restriction constrains the placement of the deployed VM: if the IPv6 interface has the default route, then the IPv4 doesn’t, which means that the VM must be deployed on the same subnet as the BOSH Director in order to communicate with it (the BOSH Director only communicates via IPv4, although we are actively working to change that).
On the other hand, if the IPv4 interface of the deployed VM has the default route, then the IPv6 interface doesn’t, which limits the usefulness of having a VM with an IPv6 address (the impetus to use IPv6 is driven by IPv4 address exhaustion, specifically routable addresses, and an IPv6 interface with no IPv6 route is not routable, and offers little value over an IPv4 address).
However, all is not lost: the IPv6 interface of the deployed VM may acquire an IPv6 route via router advertisements. [Router Advertisements] That means it’s possible to deploy a VM with both IPv4 and IPv6 default routes.
[why no abbreviations?] The BOSH Director codebase represents IPv6 addresses (in most cases, such as the internal database) as strings, and many manipulations will fail if abbreviated IPv6 addresses were used (e.g. “does this IPv6 address fall in this range?").
Although we’d like to have the capability to use abbreviated IPv6 addresses, and that may be a direction we take longer term, in the short term we must use fully-expanded IPv6 addresses. They are but a minor inconvenience to manifest writers.
[Router Advertisements] IPv6’s Neighbor Discovery Protocol’s (ND’s) Router Advertisements allow for the discovery of IPv6 routes within an IPv6 subnet. Unfortunately, they may also be used to enable man-in-the-middle attacks (Infoblox has a blog post describing the security issues).
The BOSH agent enables the acceptance of router advertisements when an IPv6 is
assigned to the VM (
bosh-agent source code:
which undoes the default settings of the
(which disable router advertisements).
We plan to disable the acceptance of IPv6’s ND’s router advertisements and
to replace it with another mechanism to allow both the IPv4 and IPv6 to have
default routes (one idea we have been considering is a new property,
Similarly, we plan to disable ICMPv6 Redirects on IPv6-enabled VMs in the future.
Use caution when connecting to a VM that has acquired an IPv6 address on its
IPv4 interface via SLAAC: having an IPv6 address on both interfaces of the
deployed VM may cause odd networking behavior. For example, when using
bosh ssh to connect to the IPv6 interface (i.e. the VM’s “far” interface) of the web
server VM from a workstation on the same subnet (same VLAN) as the VM’s IPv4
interface, and if the VM has acquired an IPv6 address on its IPv4 interface
(i.e. the VM’s “near” interface), then the
bosh ssh sessions will disconnect
(TCP RESET) within 60 seconds. A workaround would be to ssh to the IPv4
interface or the “near” IPv6 interface.
Corrections & Updates
Trimmed Gotchas section; moved excessive detail into Footnotes section
Created an Acknowledgements section
Neighbor Discovery Protocol is abbreviated “ND”, not “NP”