juju deploy --to lxd:X on manually provisioned host will always use lxdbr0

[Andrew McLeod]

This bug appears to be related to https://bugs.launchpad.net/juju/+bug/1575676

Juju 2.0.2
Ubuntu 16.04
arch s390x

Summary: I can't use a bridge other than lxdbr0 (and any config changes I make to lxdbr0 are ignored) when I attempt to deploy a charm to an lxd container on a manually provisioned machine.

Detail:

In this scenario I have manually bootstrapped an s390x environment, but assume this would apply to any architecture. Additional s390x hosts are then manually provisioned (add-machine ssh:ubuntu@ip)

Once the juju agents are reported as 'started', deploy the ubuntu charm to lxd containers on those hosts:

juju deploy ubuntu --to lxd:0 (and subsequently add-unit --to lxd:?)

The LXD containers then acquire IP addresses from lxd-bridge process / dnsmasq on the host, but I want these containers to acquire IP addresses from a DHCP server on the network (i.e. I want them to use a different bridge interface).

Regardless of the changes I make to the profile, either via "lxc profile edit default" or by modifying /etc/default/lxd-bridge, the juju-deployed containers will have virtual interfaces attached to lxdbr0:

modify /etc/default/lxd-bridge:

USE_LXD_BRIDGE="false"
LXD_BRIDGE="notlxdbr0"

directly modify and show profile:

 sudo lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: notlxdbr0
    type: nic

lxd-bridge stopped, lxd restarted

juju add-unit -n1 ubuntu --to lxd:13

| juju-e059b4-13-lxd-1 | RUNNING | | | PERSISTENT | 0 |

sudo lxc info juju-e059b4-13-lxd-1
Name: juju-e059b4-13-lxd-1
Remote: unix:/var/lib/lxd/unix.socket
Architecture: s390x
Created: 2017/01/10 05:32 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 27453
Ips:
  eth0: inet6 fe80::216:3eff:fe11:6aef vethRIXQUA
  lo: inet 127.0.0.1
  lo: inet6 ::1

ubuntu@10-13-3-10:/var/log/juju$ brctl show
bridge name bridge id STP enabled interfaces
eth0 8000.000000000000 no
lxdbr0 8000.fe9da9f9e7b4 no vethEYO2WE
       vethRIXQUA
notlxdbr0 8000.020000c15bcc no encc000.2893
ubuntu@10-13-3-10:/var/log/juju$

manually deployed containers use the correct bridge:

sudo lxc launch ubuntu
The local image 'ubuntu' couldn't be found, trying 'ubuntu:' instead.
Creating thorough-coyote
Starting thorough-coyote

| thorough-coyote | RUNNING | 10.13.3.234 (eth0) | | PERSISTENT | 0 |

$ sudo lxc info thorough-coyote
Name: thorough-coyote
Remote: unix:/var/lib/lxd/unix.socket
Architecture: s390x
Created: 2017/01/10 05:41 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 28839
Ips:
  eth0: inet 10.13.3.234 vethUO1MBH
  eth0: inet6 fe80::216:3eff:feda:ec7e vethUO1MBH
  lo: inet 127.0.0.1
  lo: inet6 ::1

$ brctl show
bridge name bridge id STP enabled interfaces
eth0 8000.000000000000 no
lxdbr0 8000.fe9da9f9e7b4 no vethEYO2WE
       vethRIXQUA
notlxdbr0 8000.020000c15bcc no encc000.2893
       vethUO1MBH

If I delete lxdbr0 and attempt to deploy another container with juju:
sudo ifconfig lxdbr0 down
sudo brctl delbr lxdbr0

juju add-unit -n1 ubuntu --to lxd:13

It fails, agent state down, and the following appears in the logs:

...
https://pastebin.canonical.com/175421/
...

I have also attempted to manually pre-configure lxdbr0, but /etc/default/lxd-bridge is overwritten by juju when "juju deploy" or "add-unit" is executed. I am not able to chattr +i this file as this causes the deploy/add-unit to fail with an error about not being able to create the file (it retries infinitely).

[Ryan Beisner (1chb1n)]

The following related bug is really the root cause here. If it didn't exist, we wouldn't have to try to munge the bridges ahead of Juju:

"manual provider lxc/lxd units are behind NAT, fail by default"
https://bugs.launchpad.net/juju/+bug/1614364

[Anastasia (anastasia-macmood)]

Since bug # 1614364 is considered Critical for 2.1.1 and it's the root cause as per comment # 1, I am removing this bug from 2.1 milestone.

We will come to re-assess whether it needs to be fixed passed 2.1.1.

[Frank Heimes (fheimes)]

Lowering importance - based on discussion in mail thread ...
(As already stated - this depends heavily on LP 1614364.)

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.