[No Network] ifupdown/bridge-utils & udev/bridge-utils race condition preventing the addition of non-pre-up ports to a bridge

[orbisvicis]

Problem:

I have configured a bridge (br0) between a physical port (eth0) and a tap device (tap0) using the ifupdown package. About 75% of the time eth0 fails to be added to br0 during boot, leaving networking inoperable. This suggests a race condition between ifupdown/bridge-utils and some other component which I believe to be udev/bridge-utils.

Layout:

Files involved:
/etc/network/interfaces
/etc/network/if-pre-up.d/bridge
/lib/bridge-utils/bridge-utils.sh
/var/log/upstart/networking.log
/var/log/upstart/network-interface-br0.log
/lib/udev/bridge-network-interface
/lib/udev/rules.d/40-bridge-network-interface.rules
/etc/default/bridge-utils

This rules out conflict between ifupdown's sysvinit and upstart integration:
$ find /etc/rc?.d/ -iname '*network*'
/etc/rc0.d/S35networking
/etc/rc6.d/S35networking

$ find /etc/init* -iname '*network*' | sort
/etc/init.d/networking
/etc/init/networking.conf
/etc/init/network-interface.conf
/etc/init/network-interface-container.conf
/etc/init/network-interface-security.conf

This rules out conflict between ifupdown/bridge-control and other ifupdown scripts
...:/etc/network$ tree
.
+-- if-down.d
¦   +-- bind9
¦   +-- openvpn
¦   +-- postfix
¦   +-- resolvconf
¦   +-- upstart
¦   +-- wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
+-- if-post-down.d
¦   +-- avahi-daemon -> ../if-up.d/avahi-daemon
¦   +-- bridge -> /lib/bridge-utils/ifupdown.sh
¦   +-- vde2
¦   +-- wireless-tools
¦   +-- wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
+-- if-pre-up.d
¦   +-- bridge -> /lib/bridge-utils/ifupdown.sh
¦   +-- ethtool
¦   +-- vde2
¦   +-- wireless-tools
¦   +-- wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
+-- if-up.d
¦   +-- 000resolvconf
¦   +-- avahi-daemon
¦   +-- bind9
¦   +-- ethtool
¦   +-- ntpdate
¦   +-- openssh-server
¦   +-- openvpn
¦   +-- postfix
¦   +-- upstart
¦   +-- wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
+-- interfaces
+-- interfaces.d
+-- run -> /run/network

Software Versions:
Ubuntu 14.04 LTS
ifupdown 0.7.47.2ubuntu4
bridge-utils 1.5-6ubuntu2
upstart 1.12.1-0ubuntu4
udev 204-5ubuntu20

Logs:
/var/log/upstart/networking.log
Sun Apr 27 17:27:03 2014 TUN/TAP device tap0 opened
Sun Apr 27 17:27:03 2014 Persist state set to: ON
device br0 already exists; can't create bridge with the same name
run-parts: /etc/network/if-pre-up.d/bridge exited with return code 1
Internet Systems Consortium DHCP Client 4.2.4
...
Listening on LPF/br0/46:33:ab:47:a7:05
Sending on LPF/br0/46:33:ab:47:a7:05
Sending on Socket/fallback
DHCPDISCOVER on br0 to 255.255.255.255 port 67 interval 3 (xid=0x5d7de88)
...
<<Fails to obtain lease>>>

/var/log/upstart/network-interface-br0.log
ifup: interface br0 already configured

My ifupdown configuration:
/etc/init.d/networking
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet dhcp
        bridge_ports eth0 tap0
        pre-up openvpn --mktun --dev tap0

Demonstrating that "pre-up openvpn --mktun --dev tap0" triggers /lib/udev/rules.d/40-bridge-network-interface.rules, which in turn runs /lib/udev/bridge-network-interface:
$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.000129f5f026 no eth0

$ sudo ip tuntap del mode tap tap0

KERNEL[59526.262768] remove /devices/virtual/net/tap0 (net)
UDEV [59526.263236] remove /devices/virtual/net/tap0 (net)

$ pre-up openvpn --mktun --dev tap0
Mon Apr 28 10:00:32 2014 TUN/TAP device tap0 opened
Mon Apr 28 10:00:32 2014 Persist state set to: ON

KERNEL[59622.299324] add /devices/virtual/net/tap0 (net)
UDEV [59622.323388] add /devices/virtual/net/tap0 (net)

$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.000129f5f026 no eth0
                                                        tap0

Confirming the problem:
By default, BRIDGE_HOTPLUG=yes in /etc/default/bridge-utils. Once disabled, however, both interfaces (eth0, tap0) are consistently added to br0 during boot, tested across five reboots.

Note that this resolves the problem and the errors logged to /var/log/upstart/networking.log ("device br0 already exists; can't create bridge with the same name..."), however the warning logged to /var/log/upstart/network-interface-br0.log (ifup: interface br0 already configured) remains (these logfiles are appended to every startup). This is not a problem but it does suggest ifup is being called twice (once from /etc/init/network-interface.conf at least) during startup.

Discussion:
Disabling BRIDGE_HOTPLUG is only a temporary hack. I have a third interface I would like to bridge which does require hotplug support.

Possible Solution
* Disable hotplugging for virtual adapters
* Disable hotplugging for a user-defined list of interfaces
* Fix the race condition?
    Not sure how, both /etc/network/if-pre-up.d/bridge and /lib/udev/rules.d/40-bridge-network-interface.rules test for the interface's presence:
    "if ... [ ! -d /sys/class/net/$IFACE ]; then"

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Giovanni Panozzo (giox069)]

I can confirm the problem in Ubuntu 14.04.1 server: I configured a bridge for openvpn:

iface br0 inet static
 address 192.168.1.6
 netmask 255.255.255.0
 gateway 192.168.1.1
 bridge_ports eth0 tap0
 dns-nameservers 192.168.1.1
 dns-search mydomain.com
 pre-up openvpn --mktun --dev tap0
 post-down openvpn --rmtun --dev tap0

and for 50% of computer boots (20% on a slower pc) the br0 setup fails and contains only tap0.

After some debugging, here are my guesses: I found that "brctl addbr br0" is called twice: from /lib/bridge-utils/ifupdown.sh and from /lib/udev/bridge-network-interface (the second one via udev due to openvpn --mktun ???).
When ifupdown.sh is called after bridge-network-interface, "brctl addbr br0" fails and returns nonzero, so the script exits and no interface are added to br0.
When the two scripts are called in the reverse order, all seems to work fine.

I used the proposed workaround to put BRIDGE_HOTPLUG=no /etc/default/bridge-utils in and now my system is working.