Juju created LXD containers with MAAS provider do not honor the default gateway on the host

[Nobuto Murata]

$ juju version
2.2.2-xenial-amd64

maas 2.2.2

The host has 103.X.Y.Z as the default gateway, but created LXD containers on top it have 10.10.23.254 as the default gateway instead. Therefore, created LXD containers cannot reach to hosts in other network because of the unexpected route. FWIW, 10.10.23.254 is the gateway of PXE boot network.

## Host

[curtin-install-cfg.yaml]
...
  - bond_interfaces:
    - eno1
    - eno2
    id: bond0
    mac_address: 24:6e:96:3d:3a:e8
    mtu: 1500
    name: bond0
    params:
      bond-downdelay: 0
      bond-lacp-rate: fast
      bond-miimon: 100
      bond-mode: 802.3ad
      bond-updelay: 0
      bond-xmit-hash-policy: layer3+4
    subnets:
    - address: 10.10.22.110/22
      dns_nameservers: []
      type: static
    type: bond
...
  - id: bond0.201
    mtu: 1500
    name: bond0.201
    subnets:
    - address: 103.<snip>/26
      dns_nameservers: []
      gateway: 103.<snip>
      type: static
    type: vlan
    vlan_id: 201
    vlan_link: bond0

$ ip r | grep default
default via 103.<snip> dev br-bond0.201 onlink

## LXD container

[/var/lib/cloud/instance/user-data.txt]
#cloud-config
apt_mirror: ""
bootcmd:
- install -D -m 644 /dev/null '/etc/network/interfaces.templ'
- |-
  printf '%s\n' '
  auto lo {eth00_16_3e_03_fe_15} {eth00_16_3e_40_6e_90} {eth00_16_3e_fe_2f_1c}

  iface lo inet loopback
    dns-nameservers 10.10.20.21
    dns-search maas

  iface {eth00_16_3e_fe_2f_1c} inet static
    address 10.10.22.106/22
    gateway 10.10.23.254

  iface {eth00_16_3e_40_6e_90} inet static
    address <snip>/26

  iface {eth00_16_3e_03_fe_15} inet static
    address 10.10.29.12/22
    mtu 9000
  ' > '/etc/network/interfaces.templ'

$ ip r | grep default
default via 10.10.23.254 dev eth0 onlink

[John A Meinel (jameinel)]

This sounds like expected behavior based on configuration. We should not inherit the host machine gateway by default.
The container should only be created with access to the subnets that you have specified. And those subnets have configuration what the gateway will be to get out of that subnet. If *that* gateway doesn't have access to public networks, that may be by design. Even if you wanted access, you still need the 10.10.23.254 gateway in order to even get to the 103.* address for it to forward any farther.

It sounds like the actual fix is that you need to fix whatever gateway is at 10.10.23.254 to forward requests further on.

It may be that you need your containers to be in more than one space, if the space for their workloads cannot get public access, and you need that. You could do that with:
  juju deploy application --bind private-space --constraints "spaces=public-space"

[Ante Karamatić (ivoks)]

I think what Nobuto is saying is that:

  iface {eth00_16_3e_40_6e_90} inet static
    address <snip>/26

is the 103..../26 address (it's public, so he obfuscated it) in the container. So, container and machine both have an IP on the network with default gateway. However, container ends up with a default gateway from wrong space.

[Ante Karamatić (ivoks)]

John, is the information not sufficient? Talking to Tim, it's apparent that this is actually expected outcome considering that Juju doesn't know what is default gateway, doesn't ask MAAS and just sets a gateway from whatever network has it defined.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/8140