juju deployed lxd containers are missing a default gateway when configured with multiple interfaces

[Matt Rae]

using juju 2.0 beta 11 and MAAS 2.0 RC1

If a host enlisted into MAAS has more than one interface configured with a subnet, then when a container is deployed to that host, the containers are missing a default gateway.

When deploying containers to a host with a single interface configured, the default gateway exists.

Example from a container below. Also attached are screenshots of my subnets and configuration for this host in maas

ubuntu@juju-01ca8f-2-lxd-0:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:cf:09:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.2.3/24 brd 192.168.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fecf:97e/64 scope link
valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:8b:c4:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.0.197/24 brd 10.20.0.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe8b:c4d5/64 scope link
valid_lft forever preferred_lft forever
ubuntu@juju-01ca8f-2-lxd-0:~$ cat /etc/network/interfaces

auto eth0 eth1 lo

iface lo inet loopback
dns-nameservers 10.20.0.11
dns-search maas

iface eth0 inet static
address 192.168.2.3/24

iface eth1 inet static
address 10.20.0.197/24

[Matt Rae (mattrae)]

Noting that juju 2.0 beta 7 did not have this issue. The default gateway gets added to containers even using multiple interfaces. The issue appears to have been introduced with beta 8.

[James Tunnicliffe (dooferlad)]

We save the original /etc/network/interfaces in /etc/network. Could you upload it? I just want to see if the issue is that a default route was listed and we removed it, or if it never got into the LXD.

[Matt Rae (mattrae)]

Hi James, what is the name of the original file that is moved to /etc/network? Looks like /etc/network is a directory.

The issue can be reproduced by deploying a container to a host in maas that has multiple interfaces configured with subnets.

I don't have the original container but here is the information from another container i reproduced this issue in.

root@juju-747636-1-lxd-0:~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

root@juju-747636-1-lxd-0:~# cat /etc/network/interfaces

auto eth0 eth1 lo

iface lo inet loopback
  dns-nameservers 10.20.0.11
  dns-search maas

iface eth0 inet static
  address 192.168.2.3/24

iface eth1 inet static
  address 10.20.0.197/24

root@juju-747636-1-lxd-0:~# cat /etc/network
cat: /etc/network: Is a directory

root@juju-747636-1-lxd-0:~# ls /etc/network
if-down.d if-post-down.d if-pre-up.d if-up.d interfaces interfaces.d

root@juju-747636-1-lxd-0:~# ls /etc/network/interfaces.d/
50-cloud-init.cfg

root@juju-747636-1-lxd-0:~# cat /etc/network/interfaces.d/50-cloud-init.cfg
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
root@juju-747636-1-lxd-0:~#

[James Tunnicliffe (dooferlad)]

Yea, juju/juju/cloudconfig/containerinit/container_userdata.go is very broken. It insists that eth0 be the first interface and that it should have the default gateway on it.

[Andrew McDermott (frobware)]

So there is something very specific about some setups that are failing. I
just tried this and I see a gateway in the containers ENI:

ubuntu@juju-2e0e5f-0-lxd-0:~$ cat /etc/network/interfaces

auto eth0 eth1 lo

iface lo inet loopback
  dns-nameservers 10.12.20.2
  dns-search maas20

iface eth0 inet static
  address 10.12.20.4/24
  gateway 10.12.20.1

iface eth1 inet static
  address 192.168.110.3/24

On 14 July 2016 at 16:27, James Tunnicliffe <email address hidden>
wrote:

> Yea, juju/juju/cloudconfig/containerinit/container_userdata.go is very
> broken. It insists that eth0 be the first interface and that it should
> have the default gateway on it.
>
> ** Changed in: juju-core
> Assignee: (unassigned) => James Tunnicliffe (dooferlad)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1602054
>
> Title:
> juju deployed lxd containers are missing a default gateway when
> configured with multiple interfaces
>
> Status in juju-core:
> Triaged
>
> Bug description:
> using juju 2.0 beta 11 and MAAS 2.0 RC1
>
> If a host enlisted into MAAS has more than one interface configured
> with a subnet, then when a container is deployed to that host, the
> containers are missing a default gateway.
>
> When deploying containers to a host with a single interface
> configured, the default gateway exists.
>
> Example from a container below. Also attached are screenshots of my
> subnets and configuration for this host in maas
>
> ubuntu@juju-01ca8f-2-lxd-0:~$ ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
> group default qlen 1
> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> inet 127.0.0.1/8 scope host lo
> valid_lft forever preferred_lft forever
> inet6 ::1/128 scope host
> valid_lft forever preferred_lft forever
> 7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP group default qlen 1000
> link/ether 00:16:3e:cf:09:7e brd ff:ff:ff:ff:ff:ff link-netnsid 0
> inet 192.168.2.3/24 brd 192.168.2.255 scope global eth0
> valid_lft forever preferred_lft forever
> inet6 fe80::216:3eff:fecf:97e/64 scope link
> valid_lft forever preferred_lft forever
> 9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP group default qlen 1000
> link/ether 00:16:3e:8b:c4:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
> inet 10.20.0.197/24 brd 10.20.0.255 scope global eth1
> valid_lft forever preferred_lft forever
> inet6 fe80::216:3eff:fe8b:c4d5/64 scope link
> valid_lft forever preferred_lft forever
> ubuntu@juju-01ca8f-2-lxd-0:~$ cat /etc/network/interfaces
>
> auto eth0 eth1 lo
>
> iface lo inet loopback
> dns-nameservers 10.20.0.11
> dns-search maas
>
> iface eth0 inet static
> address 192.168.2.3/24
>
> iface eth1 inet static
> address 10.20.0.197/24
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju-core/+bug/1602054/+subscriptions
>

--
Andrew McDermott <email address hidden>
Juju Core Sapphire team <http://juju.ubuntu.com>

[Andrew McDermott (frobware)]

WIP (missing unit tests) - https://github.com/frobware/juju/tree/master-lp1602054

[Andrew McDermott (frobware)]

Post a conversation with Jay my patch needs reworking so that we support multiple gateways, differentiated by different metric values.

[Anastasia (anastasia-macmood)]

Andrew,

based on comment #12, it looks like you a solution. Do we have a fix proposed?

[Andrew McDermott (frobware)]

I pushed an update to my fix. Mick was going to try this today. My fix is missing unit tests.

[Andrew McDermott (frobware)]

https://github.com/frobware/juju/tree/master-lp1602054 - updated today and rebased with upstream/master.

[Andrew McDermott (frobware)]

Not sure where James has got to atm, but submitting this for review in case he's not back:

  http://reviews.vapour.ws/r/5331/

[James Tunnicliffe (dooferlad)]

http://reviews.vapour.ws/r/5333/ fixes this.

[Dimiter Naydenov (dimitern)]

Proposed fix in comment #17 is not the right one IMO. I'll take over frobware's proposed fix in comment #16, clean up a few bits and propose it anew.

[Dimiter Naydenov (dimitern)]

Fix proposed: http://reviews.vapour.ws/r/5356/