BOSH deployed to an IPv6 environment on vSphere

December 17, 2017

0. Abstract

BOSH is a VM orchestrator; a BOSH Director creates, configures, monitors, and deletes VMs. The BOSH Director interoperates with a number of IaaSes (Infrastructure as a Service), one of which is VMware vSphere, a virtualization platform. BOSH traditionally operates exclusively within the IPv4 networking space (i.e. the BOSH Director has an IPv4 address (e.g. 10.0.0.6), and the VMs which it deploys also have IPv4 addresses); however, recent changes have enabled IPv6 within the BOSH Framework.

In this blog post we show how we deployed a BOSH Director with an IPv6 address (no IPv4), and, in turn, used the BOSH Director to deploy several VMs with IPv6 addresses.

We expect this blog post to be of interest to those who plan to deploy BOSH in IPv6-enabled environments.

1. Prerequisites

Use at least the following versions:

The following must have IPv6 addresses [Hybrid]

  • the workstation from which the BOSH Director is redeployed
  • the VMware vCenter
  • the VMware ESXi host

2. Deployment Overview

[Ubuntu] We haven’t yet made changes to the bosh-agent to accommodate IPv6 on the CentOS-flavored stemcells; pull requests are welcome.

History

Enabling IPv6 on BOSH was a side project we started a year ago, grossly underestimating the amount of time required — we thought it would take a couple of weeks at most; it took over a year. The changes spanned several BOSH components: the BOSH Director (e.g commit 4a35c4b8), the BOSH agent (e.g. commit 0962dce7), the BOSH CLI (e.g. commit 0316b3a5), and BOSH deployment (e.g. commit 214ebac4).

Although both our names appeared on the commits, Dmitriy did much of the heavy lifting. The code is his. Brian contributed the IPv6-enabled vSphere infrastructure and network & kernel configuration requirements.

Gotchas

Don’t abbreviate IPv6 addresses in BOSH manifests or Cloud Configs. Don’t use double ::, don’t strip leading zeroes. As an extreme example, the loopback address (::1) should be represented as 0000:0000:0000:0000:0000:0000:0000:0001.

Don’t use large reserved IP ranges (> 1k IP addresses); they will cause bosh deploy to hang. We have cheerfully named the following Cloud Config “the fifth horseman of the BOSH apocalypse”, for no deployment to that network will ever complete.

networks:
- name: IPv6
  type: manual

  subnets:
  - range:    2601:0646:0100:69f1:0000:0000:0000:0000/64
    gateway:  2601:0646:0100:69f1:020d:b9ff:fe48:9249
    dns:
    -         2001:4860:4860:0000:0000:0000:0000:8888
    -         2001:4860:4860:0000:0000:0000:0000:8844
    # This large range will cause `bosh deploy` to hang; don't do it
    reserved: [ 2601:0646:0100:69f1:0000:0000:0000:0000-2601:0646:0100:69f1:ffff:ffff:ffff:ffff ]

Don’t upload stemcells using the bosh.io URLs (bosh upload-stemcell https://s3.amazonaws.com/bosh-core-stemcells/vsphere/bosh-stemcell-...); they won’t work (Network is unreachable); Amazon S3 doesn’t have IPv6 addresses for s3.amazonaws.com. Instead, download the stemcell locally to your workstation, then upload the stemcell from your workstation to your Director (e.g. bosh upload-stemcell stemcell.tgz).

BOSH requires the IPv6 default route to reside in the same subnet as the gateway (which is not an IPv6 requirement (often the default route is an fe80::... address), though it is an IPv4 requirement).

IPv6’s Neighbor Discovery Protocol may subvert the BOSH networking model. For example, on a multi-homed VM with both IPv4 interface and IPv6 interfaces, with IPv4 interface being set as the default gateway via BOSH, and a gateway assigned to the IPv6 via Router Advertisement, may result in non-local traffic going out both the IPv4 and IPv6 interfaces instead of solely the IPv4 interface. Some may view this as a feature.

Certain versions of the vCenter Appliance require modifying /etc/sysctl.conf to enable IPv6.

BOSH doesn’t have a concept of “dual-stack”. In other words, when it deploys a VM, the VM’s network interface can have one IP address, either IPv4 or IPv6 but not both.

BOSH won’t allocate certain addresses, e.g. “subnet zero”.

Footnotes

[Ubuntu] IPv6 only works on Ubuntu stemcells; we haven’t yet made changes to the bosh-agent to accommodate IPv6 on the CentOS-flavored stemcells. Pull requests are welcome.

[Hybrid] The BOSH director and its VMs can be deployed in a hybrid manner, with both IPv4 and IPv6 addresses. Such deployments do not always require the workstation, vCenter, and ESXi host to have IPv6 addresses. Discussions of such deployments are outside the scope of this blog post.

Previous
Diagnosing performance issues with Java legacy apps on Pivotal Cloud Foundry
Diagnosing performance issues with Java legacy apps on Pivotal Cloud Foundry

The murder mystery - of the slow process We were in the process of migrating a legacy java app to PCF. We w...

Next
All I do is VIM VIM VIM
All I do is VIM VIM VIM

I love keyboard shortcuts. Naturally, VIM would be my best friend but it was really hard to get int...