Can't bring up containers on a manually provisioned machine

[Peter Sabaini]

Manually provisioning a node onto a Xenial box succeeds:

juju add-machine --constraints "tags=infra" ssh:x@a.b.c.d

However, adding a container on top via:

juju add-machine lxd:0 --constraints spaces=space-x --series xenial

Results in:

0/lxd/0 down pending xenial unexpected response fetching machine manual:a.b.c.d: []

The controller machine-0.log has this:

2018-10-04 11:13:42 WARNING juju.apiserver.provisioner provisioninginfo.go:297 not using subnet "a.b.c.d/21" in space "space-x" for machine "0/lxd/0" provisioning: no availability zone(s) set

Note that bringing up containers on the box itself via lxc launch ubuntu:x works fine, also networking within the container seems to be ok

[Richard Harding (rharding)]

Thanks for the bug. Is this 2.4.3?

[Peter Sabaini (peter-sabaini)]

Ugh, yes indeed 2.4.3 on xenial

[Joseph Phillips (manadart)]

Did you do any manual initialisation of LXD on the new host?

[Joseph Phillips (manadart)]

Also, are there any firewalls in play here?

[Peter Sabaini (peter-sabaini)]

Haven't performed any manual initialisation, no - besides installing the lxd/lxd-client packages.
The iptables rules present are those from lxd and libvirt:

iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "managed by lxd-bridge" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "managed by lxd-bridge" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 67 -m comment --comment "managed by lxd-bridge" -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A FORWARD -o lxdbr0 -m comment --comment "managed by lxd-bridge" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "managed by lxd-bridge" -j ACCEPT
-A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A FORWARD -i virbr0 -o virbr0 -j ACCEPT
-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT

[Joseph Phillips (manadart)]

I have managed to reproduce this.

It is particular to the exact combination of attempting to provision a container on a manually provisioned machine when the provider is MAAS.

What happens is:
- The provisioner asks the provider if it supports container networking.
- MAAS says yes.
- The provisioner hands off to the provider to get a container address for the host machine.
- MAAS does not recognise the (manually provisioned) host and throws and error with the observed message - "unexpected response fetching machine..."

We should certainly return a more detailed error message.

As to how we accommodate the particular scenario needs to be determined.

[Joseph Phillips (manadart)]

I have proposed https://github.com/juju/juju/pull/9287 to address this.

[Joseph Phillips (manadart)]

Patch for 2.3: https://github.com/juju/juju/pull/9292

[Joseph Phillips (manadart)]

Landed in edge (2.5) with the merge from 2.4 under https://github.com/juju/juju/pull/9299