Understanding the way containers communicate will make your life easier in many ways. Technologies like Kubernetes and Docker abstract away details that make containers special, but can also abstract away intuition and understanding. Without that knowledge, challenges arise—educated problem-solving adds confidence to decision-making!
In this post, we will demystify containers and cover some networking basics by explaining how to create two rudimentary containers and connecting them with a virtual network so they can talk to each other. The host machine, which is the machine where the network lives, views this network as if it were completely external. So, we will connect the network to the host. We’ve also included a bonus section about connecting the network to the internet so your containers can reach Google. You do not need a Linux machine to run through the exercises.
A container can be considered synonymous with a Linux network namespace. Keep this in mind. Essentially, a container is a namespace.
Each container runtime uses a namespace differently. For example, containers in Docker get their own namespace, while in CoreOS’ rkt, groups of containers share namespaces, each of which is called a pod.
Containers are based on Linux networking, and so insights learned in either can be applied to both. You can apply these concepts to anything with Linux, including containers, VMs, computers, servers, the cloud, IoT, Raspberry Pi, and supercomputers.
When you think of networking, you might think of applications communicating over HTTP, but actually a network refers more generally to a group of objects that communicate with each other by way of their unique addresses. The point is that networking refers to things communicating with things, and not necessarily an application or a container — it could be any device.
One way to connect two containers is to create a virtual network. One way to do this is by creating two namespaces and two virtual ethernet cables. Each cable should be attached to a namespace on one side, and on the opposite end be united by a bridge, to complete the network.
Now that we have defined containers are Linux namespaces, let’s see how the two go together like pb and jelly.
If you do not have access to a Linux operating system or don’t want to use what is available, you can install Multipass to easily spin up a virtual machine (VM). It’s a decent solution for what we are doing. A lot of the commands below have sudo prepending them due to Multipass.
To install using Homebrew:
brew cask install multipass
Now spin up a VM with the name ‘linux-vm’ with Ubuntu using the following command:
multipass launch -n linux-vm
To list the VMs in Multipass, use the command
To enter into a shell in your VM, use the following:
multipass shell linux-vm
To see the interfaces on your host, enter:
ip link list
This should print out the devices that are available; any networking device that has a driver loaded can be classified as an available device. In the output, you might see devices like
ip link command will also output two lines for each device, the link status, and the characteristics.
A network namespace is another copy of the network stack, with its own routes, firewall rules, and network devices. A process inherits its network namespace from its parent by default.
Let’s create two network namespaces:
sudo ip netns add pb sudo ip netns add jelly
Once they are added you can view them with
ip netns list.
You now have a very simple container that can’t do much at the moment. To get more functionality you can connect the namespaces using a veth device.
A veth device is a virtual ethernet device that you can think of as a real ethernet cable connecting two other devices. Virtual ethernet devices act as tunnels between network namespaces. They create a bridge to a physical network device in another namespace. Virtual ethernets can also be used as standalone network devices as well.
Veth devices are always created in interconnected pairs where packets transmitted on one device in the pair are immediately received on another device. When either device is down, the link state of the pair is down as well.
In our example, we are creating two veth pairs. The jelly namespace will connect to the veth-jelly end of the cable, and the other cable end should attach to a bridge that will create the network for our namespaces. We create the same cable and connect it to the bridge on the pb side.
To create both veth pairs, use the command:
sudo ip link add veth-pb type veth peer name bread-pb-veth sudo ip link add veth-jelly type veth peer name bread-j-veth
Now when you look at the devices you will see your veth pairs on the host network.
ip link list
Now that we have a veth pair in the host namespace, let’s move the jelly and pb sides of the cables out into the jelly and pb namespaces.
sudo ip link set veth-pb netns pb sudo ip link set veth-jelly netns jelly
If you check
ip link list you will no longer find
veth-jelly since they aren’t in the host namespace.
To see the ends of the cable you created, run the
ip link command within the namespaces:
sudo ip netns exec jelly \ ip link sudo ip netns exec pb \ ip link
Since we keep using
ip commands, you might be wondering what the IPv4 addresses are. Let’s quickly retrace our steps to make sense of it all.
You began by creating two namespaces, then a virtual ethernet cable. Then you connected the devices to the cable.
Now, in order to create an IP address for the jelly namespace, use the
ip address add command for the device jelly:
sudo ip netns exec jelly \ ip address add 192.168.0.55/24 dev veth-jelly sudo ip netns exec jelly \ ip link set veth-jelly up
Similarly, assign an address to pb as well:
sudo ip netns exec pb \ ip address add 192.168.0.56/24 dev veth-pb sudo ip netns exec pb \ ip link set veth-pb up
Check the interfaces in the namespaces. Verify if they are correct with the upcoming command. You are looking for the interfaces that you created:
Make sure they have the IP addresses that you set.
sudo ip netns exec jelly \ ip addr sudo ip netns exec pb \ ip addr
You now have both the IPs and interfaces set, but you can’t establish communication with them. That’s because there’s no interface in the default namespace that can send the traffic to those namespaces, and we didn’t configure addresses to the other side of the veth pairs or configure a bridge device. But with the creation of the bridge device, we’ll be able to provide the necessary routing to properly form the network.
So now, create your bridge. I called mine brd1, for the bread needed for our pb and jelly sandwich.
sudo ip link add name brd1 type bridge # set the interface sudo ip link set brd1 up
To check if the device was created, execute
With your bridge device set, it’s time to connect the bridge side of the veth cable to the bridge.
sudo ip link set bread-pb-veth up sudo ip link set bread-j-veth up
You can add the bread veth interfaces to the bridge by setting the bridge device as their master.
sudo ip link set bread-pb-veth master brd1 sudo ip link set bread-j-veth master brd1
bridge link show brd1
Try pinging pb’s IP 192.168.0.56 from the jelly namespace. Use
ctrl + C to exit.
sudo ip netns exec jelly \ ping 192.168.0.56
You have now united pb and jelly, the two “containers” you created.
If you try to
ping 192.168.0.56 from the host it won’t work because the namespace doesn’t have visibility to the global scope. As far as the namespaces are concerned, they are isolated on their own machine.
In order to access the namespaces directly from the host namespace, set the address of the bridge device bread
sudo ip addr add 192.168.0.60/24 brd + dev brd1
Now you can ping the namespaces directly from the host!
If you try to reach the internet from one of the namespaces, however, it will not work.
ip netns exec pb \ ping 22.214.171.124
The connection to the internet doesn’t work because we haven’t set a default gateway. The default gateway is the node you send every network packet that doesn’t have an explicit route. Without a default gateway, your network doesn’t know what to do with the packets it receives.
First, add a default gateway to the bridge:
sudo ip -all netns exec \ ip route add default via 192.168.0.60
Now check how the routing table looks inside your namespaces:
sudo ip netns exec jelly \ ip route sudo ip netns exec pb \ ip route
You can now connect to the internet, but can’t send or receive packets. To receive packets, configure Network Address Translation (NAT) with Masquerade. Masquerading allows machines to invisibly access the Internet via the Masquerade gateway, whereas a NAT can hide private addresses from the internet.
Lets add a new
iptables rule in the POSTROUTING chain of the NAT table to receive packets.
sudo iptables \ -t nat \ -A POSTROUTING \ -s 192.168.0.60/24 \ -j MASQUERADE
Here is a breakdown of the above flag:
Now enable packet forwarding with IPv4 ip forwarding:
sudo sysctl -w net.ipv4.ip_forward=1
And now try to reach the internet from one of your namespaces:
sudo ip netns exec jelly \ ping 126.96.36.199
You should now have complete connectivity. The host can direct traffic to a namespace, and therefore microservices within, and vice-versa. And applications in the namespace can now communicate with each other and access the internet.
You were able to create two containers and a virtual network from scratch! You have gained a more intuitive and intimate understanding of containers and Linux networking concepts, tools, and commands. Creating a container was as easy as one command
ip netns add pb to create your namespace.
Keep a lookout on the VMware Tanzu Developer website for upcoming articles. A great resource to learn more about container networking is the Container Network Interface, where you can learn how the community is standardizing container networking standards. And if you are inspired you can contribute or create your own plugin!
If you just want to run a script and check the status after or get stuck check out this script.