juju created bridge didn't include parent device in running config

[Jason Hobbs]

When deploying openstack on top of MAAS, juju created a bridge on a baremetal system. That bridge did not include the parent in the running config, as shown by brctl show:

http://paste.ubuntu.com/25874014/

The parent was written in /etc/network/interfaces, just not in the running br-eth1.2733. This caused the interface to not work.

Here is /etc/network/interfaces
http://paste.ubuntu.com/25874036/

This is with juju 2.2.6.

[Jason Hobbs (jason-hobbs)]

Logs from the run.

[John A Meinel (jameinel)]

Juju does not directly drive brctl. We specifically rely on ifup to convert
/e/n/i into the running network configuration. (We chose that route so that
we can know things will also be working correctly on a reboot.)

Is this reproducible, or a one time issue?
If you're on that machine and you issue:
 ifdown -a
 ifup -a

Does it come back correctly?
Does the machine come up correctly when rebooted?

I would be surprised if there was a race in us writing the file and running
ifup.
Was this bridge added at a different time than the rest of the bridges? (Is
it a case where we aren't restarting all of the devices and something
missing with ifup?)

Is it something silly like we're actually missing a trailing '\n' which is
causing ifup to ignore the last line?

On Thu, Nov 2, 2017 at 9:05 PM, Jason Hobbs <email address hidden>
wrote:

> Public bug reported:
>
> When deploying openstack on top of MAAS, juju created a bridge on a
> baremetal system. That bridge did not include the parent in the running
> config, as shown by brctl show:
>
> http://paste.ubuntu.com/25874014/
>
> The parent was written in /etc/network/interfaces, just not in the
> running br-eth1.2733. This caused the interface to not work.
>
> Here is /etc/network/interfaces
> http://paste.ubuntu.com/25874036/
>
> This is with juju 2.2.6.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: cdo-qa cdo-qa-blocker foundations-blocker
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1729651
>
> Title:
> juju created bridge didn't include parent device in running config
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1729651/+subscriptions
>

[Witold Krecicki (wpk)]

Could you paste the original /e/n/i ? (it should be saved somewhere in /etc/network.

[Jason Hobbs (jason-hobbs)]

I don't have the original /e/n/i from that run, here are the files from a current run on the same machine with the same config (but in this case the bridge is setup properly).

ubuntu@geodude:/etc/network$ pastebinit interfaces
http://paste.ubuntu.com/25879848/
ubuntu@geodude:/etc/network$ pastebinit interfaces.backup-1509717993
http://paste.ubuntu.com/25879849/
ubuntu@geodude:/etc/network$ pastebinit interfaces.backup-1509718374
http://paste.ubuntu.com/25879850/
ubuntu@geodude:/etc/network$ pastebinit interfaces.new
http://paste.ubuntu.com/25879851/

[Jason Hobbs (jason-hobbs)]

ifdown/ifup the bridge gets setup properly with the parent interface. I'm not sure how many times this has happened - this is the first time we've positively identified this.

[Jason Hobbs (jason-hobbs)]

We hit this again last night. The attached crashdump includes /etc/network.

[Jason Hobbs (jason-hobbs)]

re comment #7, it was machine 0 that had the issue, with br-bond0.2736.

http://paste.ubuntu.com/25955625/

[Jason Hobbs (jason-hobbs)]

Answering John's questions more thoroughly.

On Fri, Nov 3, 2017 at 12:33 AM, John A Meinel <email address hidden>
wrote:

> Juju does not directly drive brctl. We specifically rely on ifup to convert
> /e/n/i into the running network configuration. (We chose that route so that
> we can know things will also be working correctly on a reboot.)
>
> Is this reproducible, or a one time issue?
>

It's happened twice in 10 days now, but we can't reproduce it on demand.

> If you're on that machine and you issue:
> ifdown -a
> ifup -a
>
> Does it come back correctly?
>

Yes

> Does the machine come up correctly when rebooted?

Yes

> I would be surprised if there was a race in us writing the file and running
> ifup.
> Was this bridge added at a different time than the rest of the bridges? (Is
> it a case where we aren't restarting all of the devices and something
> missing with ifup?)
>

juju adds the bridge. I assume the logs contain info on when juju is
adding the bridge. We're deploying one bundle, all at once, using 'juju
deploy ./bundle.yaml'.

> Is it something silly like we're actually missing a trailing '\n' which is
> causing ifup to ignore the last line?
>

Doesn't appear to be, as it works on reboot and ifup/ifdown.

>
>
> On Thu, Nov 2, 2017 at 9:05 PM, Jason Hobbs <email address hidden>
> wrote:
>
> > Public bug reported:
> >
> > When deploying openstack on top of MAAS, juju created a bridge on a
> > baremetal system. That bridge did not include the parent in the running
> > config, as shown by brctl show:
> >
> > http://paste.ubuntu.com/25874014/
> >
> > The parent was written in /etc/network/interfaces, just not in the
> > running br-eth1.2733. This caused the interface to not work.
> >
> > Here is /etc/network/interfaces
> > http://paste.ubuntu.com/25874036/
> >
> > This is with juju 2.2.6.
> >
> > ** Affects: juju
> > Importance: Undecided
> > Status: New
> >
> >
> > ** Tags: cdo-qa cdo-qa-blocker foundations-blocker
> >
> > --
> > You received this bug notification because you are subscribed to juju.
> > Matching subscriptions: juju bugs
> > https://bugs.launchpad.net/bugs/1729651
> >
> > Title:
> > juju created bridge didn't include parent device in running config
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/juju/+bug/1729651/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729651
>
> Title:
> juju created bridge didn't include parent device in running config
>
> Status in juju:
> New
>
> Bug description:
> When deploying openstack on top of MAAS, juju created a bridge on a
> baremetal system. That bridge did not include the parent in the
> running config, as shown by brctl show:
>
> http://paste.ubuntu.com/25874014/
>
> The parent was written in /etc/network/interfaces, just not in the
> running br-eth1.2733. This caused the interface to not work.
>
> Here is /etc/network/interfaces
> http://paste.ubuntu.com/25874036/
>
> This is with juju 2.2.6.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1729...

[Witold Krecicki (wpk)]

My suggestion would be to try larger net-bond-reconfigure-delay - set it to 30, check if the problem persists.

[Jason Hobbs (jason-hobbs)]

That's not the problem; it happens with both bonded and non bonded
interfaces. the first case i reported was non bonded - br-eth1.2733.

On Tue, Nov 14, 2017 at 1:35 PM, Witold Krecicki <<email address hidden>
> wrote:

> My suggestion would be to try larger net-bond-reconfigure-delay - set it
> to 30, check if the problem persists.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729651
>
> Title:
> juju created bridge didn't include parent device in running config
>
> Status in juju:
> New
>
> Bug description:
> When deploying openstack on top of MAAS, juju created a bridge on a
> baremetal system. That bridge did not include the parent in the
> running config, as shown by brctl show:
>
> http://paste.ubuntu.com/25874014/
>
> The parent was written in /etc/network/interfaces, just not in the
> running br-eth1.2733. This caused the interface to not work.
>
> Here is /etc/network/interfaces
> http://paste.ubuntu.com/25874036/
>
> This is with juju 2.2.6.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1729651/+subscriptions
>

[John A Meinel (jameinel)]

We can investigate, but as near as we can determine this seems to be a bug in ifdown/ifup if it isn't actually doing what is written in /e/n/i

[Joseph Phillips (manadart)]

This area of the logic has had wholesale rewrites since this time.