Bonded interfaces don't come down with a ifdown -a

[Bryan Quigley]

1) Precise -> Trusty
2) 1.1.0_19ubuntu5
3) a. Set up new interfaces config with a simple bonded config (you don't need eth0):
auto eth0
iface eth0 inet dhcp

auto eth1
iface eth1 inet manual
 bond-master bond0
 bond-primary eth1

auto eth2
iface eth2 inet manual
  bond-master bond0

auto bond0
iface bond0 inet static
 address 192.168.122.99
 gateway 192.168.122.1
 netmask 255.255.255.0
 bond-mode active-backup
 bond-miimon 100
 bond-slaves none

b. Restart the machine to bring up in a known state (wait an extra few minutes to make sure the interfaces aren't still waiting)
c. Note what interfaces come up correctly via ifconfig
d. Run ifdown -a. This should bring down all interfaces....

4) Ifdown -a doesn't bring down bond0

Observations:
  - if we're using dhcp for the bond0 interface this works
  - if there isn't an extra eth0 device (as described above) this works

[Chris J Arges (arges)]

This is also reproducible on Trusty, if I use the same configuration above and execute the following:
$ sudo ifdown bond0
I get:
ifdown: interface bond0 not configured

I straced the command and see it reads /run/network/ifstate. However the file doesn't have bond0 and therefore we get the error. This is even if ifconfig / ip addr show the device as UP and having a proper ipv4 address.

[Chris J Arges (arges)]

I also noticed that ifup behaves differently for bond0 than eth0 for example:

$ sudo ifup bond0
sh: echo: I/O error
Waiting for a slave to join bond0 (will timeout after 60s)
RTNETLINK answers: File exists
Failed to bring up bond0.

$ sudo ifup eth0
ifup: interface eth0 already configured

[Chris J Arges (arges)]

After a reboot I see the following:
$ ls /run/network/
ifenslave.bond0 ifstate ifup.eth0 ifup.eth1 ifup.eth2 ifup.lo

This causes 'ifdown bond0' to fail. If we workaround it by doing:
$ sudo touch /run/network/ifup.bond0
$ echo 'bond0=bond0' | sudo tee /run/network/ifstate

We can run this now:
$ sudo ifdown bond0
RTNETLINK answers: No such process

And ip addr shows eth1/eth2 down and bond0 not present at all.

This makes me suspect that the files in /run/network aren't being created properly in the ifup -a case when we have a bond.

[Bryan Quigley (bryanquigley)]

I can't reproduce this bug on trusty anymore... in any case.. the ifupdown package from this bug claims to fix it! - https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1294807

"ifupdown (0.7.47.2ubuntu2~ppa1) trusty; urgency=medium

  * Allow setting the MTU and HWADDR on manual interfaces.
  * The above change also means that manual interfaces will now be
    brought up and down (as many users expected until now).
 -- Stephane Graber <email address hidden> Wed, 19 Mar 2014 18:04:49 -0400"

[Chris J Arges (arges)]

I can _still_ hit this issue with 0.7.47.2ubuntu2~ppa1. In the above case bond0 is a static interface not a manual interface.

[Stéphane Graber (stgraber)]

Took a little while to figure out but the config in this bug report is actually invalid.

You can't have two interfaces defining a default gateway with the same metric, which is exactly what you are doing when mixing an "inet dhcp" and "inet static + gateway entry".

That error was unfortunately only printed in /var/log/upstart/network-interface-bond0.log so wasn't terribly easy to spot (and there isn't much I can do to make it more visible...).

Now depending on which of ifupdown or dhclient was fastest you'd either get a working setup without the DHCP provided route or you would get a system with the DHCP route but a misconfigured bond.

[Stéphane Graber (stgraber)]

I'm checking whether we can make the original ifup fail too in one way or another, not sure how much clearer things would be if that happened though.

[Chris J Arges (arges)]

So if a 'metric 10' line is added to the bond0 stanza this works as expected. What happens is 'ip addr add ...' is called adding the default gateway and we get a 'RTNETLINK answers: File exists' which puts ifupdown into a bad state.

[Chris J Arges (arges)]

Also: The real fix would be to ensure that multiple gateways aren't defined in your networking configuration.

[Stéphane Graber (stgraber)]

Hmm, so there's no good way to trigger an action on failure of a call done by ifup itself...

I also thought of having the slave interface detect the condition from the pre-up script, but that's not really possible because bond0 won't hit the IP config stage until the slave is fully configured at which point we can't have the slave abort...

So I'm afraid this is one of those cases where documentation is our best way forward for 14.04...

[Mike Reid (akfypz-mike-nw2wga)]

Thank you, Chris.

Your comment from 2014-03-20 regarding "multiple gateways" finally helped me to resolve our dreaded startup delays on trusty (14.04.3 LTS) that appeared the moment we activated our bonded interface. Took lots of digging, appreciate you documenting the required fix.