newly created LXD container has zero network devices

[Andrew McDermott]

Using:

$ juju version
2.0-beta4-xenial-amd64

I bootstrap (xenial) on a node that has an interface (eth0) plus a number of VLANs:

$ maas maas nodes list | grep 192.168 | grep cidr | sort -u
    "cidr": "192.168.10.0/24",
    "cidr": "192.168.11.0/24",
    "cidr": "192.168.12.0/24",
    "cidr": "192.168.16.0/21",
    "cidr": "192.168.24.0/21",

$ juju add-machine lxd:0

but the network profile for the container is missing all network devices.

$ sudo lxc profile show juju-machine-0-lxd-0-network

name: juju-machine-0-lxd-0-network
config: {}
description: ""
devices: {}

If instead you bootstrap and then immediately reboot the bootstrap node before adding the container you see the following (correct) configuration:

$ sudo lxc profile show juju-machine-0-lxd-0-network

name: juju-machine-0-lxd-0-network
config: {}
description: ""
devices:
  eth0:
    hwaddr: 00:16:3e:11:ab:ee
    mtu: "1500"
    name: eth0
    nictype: bridged
    parent: br-eth0
    type: nic
  eth1:
    hwaddr: 00:16:3e:08:4b:d8
    mtu: "1500"
    name: eth1
    nictype: bridged
    parent: br-eth0.10
    type: nic
  eth2:
    hwaddr: 00:16:3e:2b:76:cf
    mtu: "1500"
    name: eth2
    nictype: bridged
    parent: br-eth0.11
    type: nic
  eth3:
    hwaddr: 00:16:3e:32:c5:3f
    mtu: "1500"
    name: eth3
    nictype: bridged
    parent: br-eth0.12
    type: nic
  eth4:
    hwaddr: 00:16:3e:a4:52:9a
    mtu: "1500"
    name: eth4
    nictype: bridged
    parent: br-eth0.13
    type: nic
  eth5:
    hwaddr: 00:16:3e:89:ed:fb
    mtu: "1500"
    name: eth5
    nictype: bridged
    parent: br-eth0.14
    type: nic

This is potentially timing related because very occasionally I do see one network inteface added, typically the eth0 device.

[Andrew McDermott (frobware)]

Raising this to critical as the OpenStack folk have already bumped into this issue deploying charms via LXD.

<jamespage> frobware, lxd containers on machine 0 appear to be OK
<jamespage> just not on 1 and 2
<frobware> jamespage: aha.
<frobware> jamespage: you have run into https://bugs.launchpad.net/juju-core/+bug/1564395
<mup> Bug #1564395: newly created LXD container has zero network devices <bootstrap> <network> <juju-core:New> <https://launchpad.net/bugs/1564395>
<frobware> jamespage: which I raised about 30 mins ago
<frobware> jamespage: the containers on 1 have no network devices - http://pastebin.ubuntu.com/15567781/

[Andrew McDermott (frobware)]

A workaround is to reboot the node before adding containers.

[Andrew McDermott (frobware)]

I moved to tip of master today (54c7168aa14b4b94c1c6a6c87aa697aa9ae815bd) and the reboot workaround no longer works.

I still get the LXD profile generated correctly (post the node's reboot) but now the containers /e/n/i is no longer correct:

jubuntu@maas19-node2:~$ sudo su
root@maas19-node2:/home/ubuntu# cat /var/lib/lxd/containers/juju-machine-0-lxd-1/rootfs/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# Source interfaces
# Please check /etc/network/interfaces.d before changing this file
# as interfaces may have been defined in /etc/network/interfaces.d
# NOTE: the primary ethernet device is defined in
# /etc/network/interfaces.d/eth0
# See LP: #1262951
source /etc/network/interfaces.d/*.cfg

This is very broken now.

[Andrew McDermott (frobware)]

In the juju log we still see the network config get generated:

2016-04-05 12:58:04 DEBUG juju.cloudconfig.containerinit container_userdata.go:142 generated network config from []network.InterfaceInfo{network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"192.168.10.0/24", NetworkName:"", ProviderId:"805", ProviderSubnetId:"4", ProviderSpaceId:"", ProviderVLANId:"5006", ProviderAddressId:"2333", AvailabilityZones:[]string(nil), VLANTag:10, InterfaceName:"eth0", ParentInterfaceName:"br-eth0.10", InterfaceType:"ethernet", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:local-cloud:192.168.10.103, DNSServers:[]network.Address{}, MTU:1500, DNSSearchDomains:[]string(nil), GatewayAddress:local-cloud:192.168.10.1, ExtraConfig:map[string]string(nil)}, network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"192.168.12.0/24", NetworkName:"", ProviderId:"806", ProviderSubnetId:"6", ProviderSpaceId:"", ProviderVLANId:"5008", ProviderAddressId:"2326", AvailabilityZones:[]string(nil), VLANTag:12, InterfaceName:"eth1", ParentInterfaceName:"br-eth0.12", InterfaceType:"ethernet", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:local-cloud:192.168.12.103, DNSServers:[]network.Address{}, MTU:1500, DNSSearchDomains:[]string(nil), GatewayAddress:, ExtraConfig:map[string]string(nil)}, network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"192.168.16.0/21", NetworkName:"", ProviderId:"807", ProviderSubnetId:"10", ProviderSpaceId:"", ProviderVLANId:"5012", ProviderAddressId:"2328", AvailabilityZones:[]string(nil), VLANTag:13, InterfaceName:"eth2", ParentInterfaceName:"br-eth0.13", InterfaceType:"ethernet", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:local-cloud:192.168.16.101, DNSServers:[]network.Address{}, MTU:1500, DNSSearchDomains:[]string(nil), GatewayAddress:, ExtraConfig:map[string]string(nil)}, network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"192.168.24.0/21", NetworkName:"", ProviderId:"808", ProviderSubnetId:"11", ProviderSpaceId:"", ProviderVLANId:"5013", ProviderAddressId:"2330", AvailabilityZones:[]string(nil), VLANTag:14, InterfaceName:"eth3", ParentInterfaceName:"br-eth0.14", InterfaceType:"ethernet", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:local-cloud:192.168.24.101, DNSServers:[]network.Address{}, MTU:1500, DNSSearchDomains:[]string(nil), GatewayAddress:, ExtraConfig:map[string]string(nil)}, network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"192.168.32.0/21", NetworkName:"", ProviderId:"809", ProviderSubnetId:"12", ProviderSpaceId:"", ProviderVLANId:"5015", ProviderAddressId:"2332", AvailabilityZones:[]string(nil), VLANTag:15, InterfaceName:"eth4", ParentInterfaceName:"br-eth0.15", InterfaceType:"ethernet", Disabled:false, NoAutoStart:false, ConfigType:"static", Address:local-cloud:192.168.32.101, DNSServers:[]network.Address{}, MTU:1500, DNSSearchDomains:[]string(nil), GatewayAddress:, ExtraConfig:map[string]string(nil)}, network.InterfaceInfo{DeviceIndex:0, MACAddress:"", CIDR:"10.17.20.0/24", NetworkName:"", ProviderId:"810", ProviderSubnetId:"1", ProviderSpaceId:"", ProviderVLANId:"0", ProviderAddressId:"2335", AvailabilityZones:[]string(nil), VLA...

[Andrew McDermott (frobware)]

And if I take the config we generate from #4 and replace the containers config we now see:

ubuntu@maas19-node2:~$ sudo lxc list
+----------------------+---------+--------------------------------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------------------+---------+--------------------------------+------+------------+-----------+
| juju-machine-0-lxd-0 | RUNNING | 10.0.3.1 (lxcbr0) | | PERSISTENT | 0 |
+----------------------+---------+--------------------------------+------+------------+-----------+
| juju-machine-0-lxd-1 | RUNNING | 10.0.3.1 (lxcbr0) | | PERSISTENT | 0 |
+----------------------+---------+--------------------------------+------+------------+-----------+
| juju-machine-0-lxd-2 | RUNNING | 192.168.12.103 (eth1) | | PERSISTENT | 0 |
| | | 192.168.16.101 (eth2) | | | |
| | | 192.168.24.101 (eth3) | | | |
| | | 192.168.32.101 (eth4) | | | |
| | | 10.17.20.214 (eth5) | | | |
| | | 192.168.11.103 (eth6) | | | |
+----------------------+---------+--------------------------------+------+------------+-----------+

[Andrew McDermott (frobware)]

Out of completeness I went back to:

root@maas19-node2:/home/ubuntu# cat /var/lib/lxd/containers/juju-machine-0-lxd-1/rootfs/etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
  dns-nameservers 10.17.20.200
  pre-up ip address add 10.17.20.214/24 dev eth0 || true
  up ip route replace 10.17.20.0/24 dev eth0 || true
  down ip route del 10.17.20.0/24 dev eth0 || true
  post-down address del 10.17.20.214/24 dev eth0 || true
  up ip route replace default via 10.17.20.1 || true
  down ip route del default via 10.17.20.1 || true

auto eth1
iface eth1 inet manual
  pre-up ip address add 192.168.10.103/24 dev eth1 || true
  up ip route replace 192.168.10.0/24 dev eth1 || true
  down ip route del 192.168.10.0/24 dev eth1 || true
  post-down address del 192.168.10.103/24 dev eth1 || true

auto eth2
iface eth2 inet manual
  pre-up ip address add 192.168.11.103/24 dev eth2 || true
  up ip route replace 192.168.11.0/24 dev eth2 || true
  down ip route del 192.168.11.0/24 dev eth2 || true
  post-down address del 192.168.11.103/24 dev eth2 || true

auto eth3
iface eth3 inet manual
  pre-up ip address add 192.168.12.103/24 dev eth3 || true
  up ip route replace 192.168.12.0/24 dev eth3 || true
  down ip route del 192.168.12.0/24 dev eth3 || true
  post-down address del 192.168.12.103/24 dev eth3 || true

auto eth4
iface eth4 inet manual
  pre-up ip address add 192.168.16.101/21 dev eth4 || true
  up ip route replace 192.168.16.0/21 dev eth4 || true
  down ip route del 192.168.16.0/21 dev eth4 || true
  post-down address del 192.168.16.101/21 dev eth4 || true

auto eth5
iface eth5 inet manual
  pre-up ip address add 192.168.24.101/21 dev eth5 || true
  up ip route replace 192.168.24.0/21 dev eth5 || true
  down ip route del 192.168.24.0/21 dev eth5 || true
  post-down address del 192.168.24.101/21 dev eth5 || true

auto eth6
iface eth6 inet manual
  pre-up ip address add 192.168.32.101/21 dev eth6 || true
  up ip route replace 192.168.32.0/21 dev eth6 || true
  down ip route del 192.168.32.0/21 dev eth6 || true
  post-down address del 192.168.32.101/21 dev eth6 || true

root@maas19-node2:/home/ubuntu# exit
ubuntu@maas19-node2:~$ logout
Connection to 10.17.20.213 closed.
aim@spicy:~/juju ((0798f54...)) (local.maas)
$ git log
commit 0798f5452273f653c561fb6d8f8543aa85ea7f5a
Merge: 965f56c 68fdac3
Author: Juju bot <email address hidden>
Date: Sat Mar 19 17:36:04 2016 +0000

    Merge pull request #4802 from dimitern/m-s-m-n-c-w-m-multi-nic-lxd

    Enable multi-NIC LXD containers on MAAS

    This is essentially https://github.com/juju/juju/pull/4789 with an minor
    fix for cloudconfig/containerinit tests. Enables multi-NIC LXD container
    provisioning on MAAS, using the same approach previously tested with
    LXC.

    Live tested on MAAS 1.9 with a mediawiki bundle deployment including
    placement to both LXC and LXD containers: http://paste.ubuntu.com/15426062/
    on both trusty and xenial.

    (Review request: http://reviews.vapour.ws/r/4244/)

and added a container to see what its /e/n/i would look like (which looks OK)

root@maas19-node2:/home/ubuntu# cat /var/lib/lxd/containers/...

[Andrew McDermott (frobware)]

The previous comment was about moving back to:

$ git log
commit 0798f5452273f653c561fb6d8f8543aa85ea7f5a
Merge: 965f56c 68fdac3
Author: Juju bot <email address hidden>
Date: Sat Mar 19 17:36:04 2016 +0000

(but cut+paste error)

and starting a container, and observing the containers /e/n/i, which appears to be as expected.

[Andrew McDermott (frobware)]

Regarding comment #3, please see IRC conversation http://pastebin.ubuntu.com/15634033/

[Dimiter Naydenov (dimitern)]

Fix proposed: https://github.com/juju/juju/pull/5045

[Dimiter Naydenov (dimitern)]

More reliable fix proposed: https://github.com/juju/juju/pull/5198