upgraded juju from 2.0.0: nodes allocation fails if any bridge created in MAAS

[Alvaro Uria]

Juju 2.0.2
MaaS 2.1.1+bzr5544-0ubuntu1 (16.04.1)

Steps to reproduce:
1) create a bridge for one or all the interfaces gathered after Commissioning phase
2) juju deploy cs:xenial/ubuntu

Node in Ready state gets allocated, and briefly moves to Deploying status to go back to Ready state. Node gets powered on ("power off" is not triggered).

If we remove all bridges created in maas, node gets deployed OK (br-$ifacename is created).

The behavior above is not documented and doesn't allow nodes to have multiple bridges (ie. br0-mgmt, with an assigned IP + br1-pub, unconfigured bridge for LXDs).

[Anastasia (anastasia-macmood)]

We track Juju 2.x issues in "juju" project. Re-targeting :)

[Andrew McDermott (frobware)]

When you say "create a bridge for one or all the interfaces gathered after Commissioning phase" is this something you are doing in the MAAS UI, or something you are doing in a custom curtin/MAAS install script?

When the machine deploys OK please could you recursively attach the contents of /etc/network/interfaces*.

[Andrew McDermott (frobware)]

Can you also run the deploy with debug:

  $ juju deploy cs:xenial/ubuntu --debug

and attach the output to the bug.

[Alvaro Uria (aluria)]

Hi Andrew. Yeah, I'm creating such bridge(s) from MAAS UI.
* I've re-tested it by creating br0 against main interface (the one selected as PXE).
* I've configured br0 as the interface was (untagged, 10.245.88.0/23, (Auto assign))

$ juju deploy cs:xenial/ubuntu ubuntu-lp1656304 --debug --constraints tags=os-cs |& tee ubuntu-lp1656304.log
10:34:54 INFO juju.cmd supercommand.go:63 running juju [2.0.0 gc go1.6.2]
10:34:54 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "deploy", "cs:xenial/ubuntu", "ubuntu-lp1656304", "--debug", "--constraints", "tags=os-cs"}
10:34:54 INFO juju.juju api.go:72 connecting to API addresses: [10.245.89.10:17070]
10:34:54 INFO juju.api apiclient.go:519 dialing "wss://10.245.89.10:17070/model/83516b3d-94d1-4391-86e6-51cd687bc69c/api"
10:34:54 INFO juju.api apiclient.go:309 connection established to "wss://10.245.89.10:17070/model/83516b3d-94d1-4391-86e6-51cd687bc69c/api"
10:34:54 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
10:34:54 DEBUG juju.cmd.juju.application deploy.go:762 cannot interpret as local bundle: bundle not found: cs:xenial/ubuntu
10:34:54 DEBUG juju.cmd.juju.application deploy.go:817 cannot interpret as local charm: file does not exist
10:34:54 DEBUG juju.cmd.juju.application deploy.go:713 cannot interpret as a redeployment of a local charm from the controller
10:34:54 DEBUG httpbakery client.go:244 client do GET https://api.jujucharms.com/charmstore/v5/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published {
10:34:55 DEBUG httpbakery client.go:246 } -> error <nil>
10:34:55 DEBUG juju.cmd.juju.application deploy.go:865 cannot interpret as charmstore bundle: (series) != "bundle"
10:34:55 DEBUG httpbakery client.go:244 client do GET https://api.jujucharms.com/charmstore/v5/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published {
10:34:55 DEBUG httpbakery client.go:246 } -> error <nil>
10:34:55 INFO juju.cmd.juju.application series_selector.go:104 with the user specified series "xenial"
10:34:55 INFO cmd cmd.go:129 Located charm "cs:ubuntu-10".
10:34:55 INFO cmd cmd.go:129 Deploying charm "cs:ubuntu-10".
10:34:55 DEBUG juju.api monitor.go:35 RPC connection died
10:34:55 INFO cmd supercommand.go:465 command finished

[Alvaro Uria (aluria)]

If I leave a node without bridges configured in MAAS, it works fine:

$ juju deploy cs:xenial/ubuntu ubuntu-nobr-lp1656304 --debug --constraints tags=os-cs |& tee ubuntu-nobr-lp1656304.log
10:33:00 INFO juju.cmd supercommand.go:63 running juju [2.0.0 gc go1.6.2]
10:33:00 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "deploy", "cs:xenial/ubuntu", "ubuntu-nobr-lp1656304", "--debug", "--constraints", "tags=os-cs"}
10:33:00 INFO juju.juju api.go:72 connecting to API addresses: [10.245.89.10:17070]
10:33:00 INFO juju.api apiclient.go:519 dialing "wss://10.245.89.10:17070/model/83516b3d-94d1-4391-86e6-51cd687bc69c/api"
10:33:00 INFO juju.api apiclient.go:309 connection established to "wss://10.245.89.10:17070/model/83516b3d-94d1-4391-86e6-51cd687bc69c/api"
10:33:00 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
10:33:00 DEBUG juju.cmd.juju.application deploy.go:762 cannot interpret as local bundle: bundle not found: cs:xenial/ubuntu
10:33:00 DEBUG juju.cmd.juju.application deploy.go:817 cannot interpret as local charm: file does not exist
10:33:00 DEBUG juju.cmd.juju.application deploy.go:713 cannot interpret as a redeployment of a local charm from the controller
10:33:00 DEBUG httpbakery client.go:244 client do GET https://api.jujucharms.com/charmstore/v5/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published {
10:33:00 DEBUG httpbakery client.go:246 } -> error <nil>
10:33:00 DEBUG juju.cmd.juju.application deploy.go:865 cannot interpret as charmstore bundle: (series) != "bundle"
10:33:00 DEBUG httpbakery client.go:244 client do GET https://api.jujucharms.com/charmstore/v5/xenial/ubuntu/meta/any?include=id&include=supported-series&include=published {
10:33:01 DEBUG httpbakery client.go:246 } -> error <nil>
10:33:01 INFO juju.cmd.juju.application series_selector.go:104 with the user specified series "xenial"
10:33:01 INFO cmd cmd.go:129 Located charm "cs:ubuntu-10".
10:33:01 INFO cmd cmd.go:129 Deploying charm "cs:ubuntu-10".
10:33:02 INFO cmd supercommand.go:465 command finished

PS: FTR, as I did "juju bootstrap bs2-stg-transient --constraints tags=bootstrap", if I don't deploy using tags (ie. tags=os-cs), "bootstrap" tag constraint will remain active.

[Andrew McDermott (frobware)]

This looks like a version issue:

 10:33:00 INFO juju.cmd supercommand.go:63 running juju [2.0.0 gc go1.6.2]

but you say at the beginning of the bug report you are running 2.0.2.

I fixed this:

 https://bugs.launchpad.net/juju/+bug/1644720

and if I go back to 2.0.0 and just bridge a single interface and bootstrap I see the errors reported in #1644720:

 2017-01-17 11:32:03 ERROR juju.worker.dependency engine.go:539 "machiner" manifold worker returned unexpected error: cannot update observed network config: cannot set link-layer device addresses of machine "0": invalid address "172.16.103.8/24": DeviceName "br0" on machine "0" not found (not found)

Please could you ensure you're actually picking up and running with juju-2.0.2.

[Anastasia (anastasia-macmood)]

As per comment above, the issue has been fix. I am marking it as such \o/

Thank you, Andy :D

[Alvaro Uria (aluria)]

Hi,

Issue is still not fixed. Juju client was 2.0.0 but after a "juju upgrade-juju", issue was still happening. However, and by Andrew advice, I also updated juju client version and still happens.

Please find machine-0.log file attached.

[Andrew McDermott (frobware)]

I suspect that this is now an upgrade-juju issue. I was not able to bootstrap with a bridge in MAAS using 2.0.0, but I was with 2.0.2.

If I start with 2.0.0, then move to 2.0.2 via upgrade-juju the problem persists.

[Alvaro Uria (aluria)]

* I bootstraped the env from scratch: https://pastebin.canonical.com/176158/
* I deployed a node with a couple of bridges configured from MAAS UI (br0, br1): https://pastebin.canonical.com/176159/
* From "juju status" perspective, unit is in "allocating" state and machine is "down".
* From MAAS UI perspective, node is powered on and in "Ready" state, after a short transition to "allocating" and back to "Ready"

Please find attached new machine-0.log.

[Tim Penhey (thumper)]

Not sure about the actual priority of this. If this continues to be a big problem, let me know.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.