IPv6 connectivity of LXC containers

[Frank Mueller]

GOAL

IPv6 subnets, like eg. 2001:db8:beef:1::/64, are routed to Juju machines. Containers running on this machine are assigned individual IPv6 addresses or sub-subnets of this subnet.

NETWORK LAYOUT

1) Host 1: 2001:db8:beef:1::/64
1a) eth0: 2001:db8:beef:1::1:1/64
1b) lxcbr0: 2001:db8:beef:1::1:2/64
1c) Container 11: 2001:db8:beef:1::2:1/64
1d) Container 12: 2001:db8:beef:1::2:2/64
2) Host 2: 2001:db8:beef:2::/64
2a) eth0: 2001:db8:beef:2::1:1/64
2b) lxcbr0: 2001:db8:beef:2::1:2/64
2c) Container 21: 2001:db8:beef:2::2:1/64
2d) Container 22: 2001:db8:beef:2::2:2/64

Other tests used the same subnet for both hosts or /112 subnets for the containers.

TEST SETUP

Variant A
- Host 1 and 2 are VMs
- Host 1 runs containers 11 and 12
- Host 2 runs containers 21 and 22

Variant B
- Host 1 and 2 are containers
- Host 1 runs sub-containers 11 and 12
- Host 2 runs sub-containers 21 and 22

STATUS

- host 1 can ping6 host 2
- host 1 can ping6 container 11 and container 12 (same on host 2)
- container 11 can ping6 container 12 (same on host 2)
- host 1 cannot (!) ping6 container 21 and container 22 (and vice versa)
- tcpdump shows, that host of receiving container receives ICMP6 neighbor solicitation, but doesn’t answer
- pinged container doesn’t see ICMP6 neighbor solicitation

NETWORK SETUP SCRIPT

-----8<--------------------
#!/bin/bash

HOST=1
CN1="sub-cn-${HOST}1"
CN2="sub-cn-${HOST}2"

NETWORK="2001:db8:beef:${HOST}"
ETH0ADDR="${NETWORK}::1:1/64"
LXCBR0ADDR="${NETWORK}::1:2/64"
LXCBR0NET="${NETWORK}::2:0/112"
ROUTE="fe80::1"

CN1ADDR="${NETWORK}::2:1/64"
CN2ADDR="${NETWORK}::2:2/64"

echo "Starting containers ..."

sudo lxc-start -n ${CN1} -d
sudo lxc-start -n ${CN2} -d

sleep 2
sudo lxc-ls -f

echo "Setting up ETH0 ... "

sudo ip -6 addr add ${ETH0ADDR} dev eth0
sudo ip -6 route add default dev eth0

echo "Setting up LXCBR0 ..."

sudo ip -6 addr add ${LXCBR0ADDR} dev lxcbr0
sudo ip -6 route add ${LXCBR0NET} dev lxcbr0

echo "Setting up container 1 ..."

sudo lxc-attach -n ${CN1} -- ip -6 addr add ${CN1ADDR} dev eth0
sudo lxc-attach -n ${CN1} -- ip -6 route add default via ${ROUTE} dev eth0

echo "Setting up container 2 ..."

sudo lxc-attach -n ${CN2} -- ip -6 addr add ${CN2ADDR} dev eth0
sudo lxc-attach -n ${CN2} -- ip -6 route add default via ${ROUTE} dev eth0

sleep 1
sudo lxc-ls -f
-----8<--------------------

[John A Meinel (jameinel)]

Just adding a link since it has some relevance for how to set up networking with LXC containers:
  http://containerops.org/2013/11/19/lxc-networking/

[Stéphane Graber (stgraber)]

Using subnets smaller than /64 is not recommended as this is likely to interfere with multicast and other IPv6 features.

Instead what you should have is some kind of connectivity on each host, then a /64 be routed to the IP of each hosts which you can then hand out to dnsmasq or radvd to distribute to your containers.

That's basically what I've got setup on all my hosts here, I've got a tweaked lxcbr0 which runs dnsmasq for both IPv4 and IPv6 using 10.0.3.x/24 for IPv4 and a routed /64 for IPv6, dnsmasq will do both DHCPv6 and RA for you.

[Jay Vosburgh (jvosburgh)]

Just capturing some conversation from the interlock call:

- The two hosts are on 2001:db8:beef:1/64 the other is 2001:db8:beef:2/64, so there is routing, but if the packets reach the destination host the routing hop should be complete, and the remaining path is a bridge hop from the receiving host to the receiving host container.

- one thought is that the bridge is failing to forward for some reason, then perhaps using OVS instead of bridge would produce different behavior

[Jay Vosburgh (jvosburgh)]

 Short version:

 This appears to be a configuration issue, and not a bug. I'm not
really an IPv6 expert, though, so I could be wrong.

 Long version:

 I set this up to test with outside of juju (I built some
containers manually), so I might be missing some juju fru fru or some
subtle aspect of IPv6, but it appears that for this configuration on a
given host:

sub-nc-21: eth0 2001:db8:beef:2::2:1/64

host of 21: eth0 2001:db8:beef:2::1:1/64

host of 21: lxcbr0 2001:db8:beef:2::1:2/64

 The bridge ports are veth devices to the containers (only); eth0
is not a member of the bridge.

 When sub-nc-21 issues a neighbor solicitation for
2001:db8:beef:2::1:2 (the ip of lxcbr0), it is correctly answered.

 However, when sub-nc-21 issues an NS for eth0 address of
2001:db8:beef:2::1:1, this is correctly not answered, because the
receiving interface, lxcbr0, has only 2001:db8:beef:2::1:2, not
[...]::1:1.

 IPv6 neighbor discovery operates differently than linux's IPv4 ARP
default behavior (in which an ARP request is replied to if any interface
has the requested address); IPv6 ND only replies if the receiving
interface has the requested address.

 The situations cited in the bug description:

- host 1 cannot (!) ping6 container 21 and container 22 (and vice versa)
- tcpdump shows, that host of receiving container receives ICMP6 neighbor solicitation, but doesn’t answer

 Are due to the same effect; host 1 issues a NS for container 21's
address, but that address is not assigned to host 2's receiving interface
(eth0), so no reply is generated.

 Linux IPv4 ARP would behave as IPv6 ND does if the arp_ignore
sysctl is set to a value or 1 or 2 instead of the default 0.

 As far as resolving this, I think that stgraber's suggestion in
comment #2 is probably the way to go.

 The network topology described here is a bit odd, in that the
bridge, the containers attached to the bridge, and the eth0 interface are
all configured for the same subnet. However, the containers cannot access
eth0 directly, and vice versa, so even if a neighbor entry is added
manually, there is no unrouted connectivity between eth0 and the
containers.

 If I add a new address to eth0, 2001:db8:beef:3::1:1/64, that is now
reachable from the containers where 2001:db8:beef:1::1:1/64 is not. This
works for me because the fe80::1/64 default route on the containers routes
through the bridge, where I added fe80::1/64 as an address and enabled
forwarding. I'm not sure where fe80::1/64 ends up in a real juju
configuration.