bridge_ports all doesn't work at boot time

[Daniel Senderowicz]

I posted the details of the problem on: http://ubuntuforums.org/showthread.php?t=1911817

/etc/network/interfaces:
auto lo
iface lo inet loopback

auto br0
iface br0 inet static
  address 192.168.1.5
  netmask 255.255.255.0
  gateway 192.168.1.1
  bridge_ports all

iface eth0 inet manual
  up ifconfig $IFACE 0.0.0.0 up
  up ip link set $IFACE promisc on
  down ip link set $IFACE promisc off
  down ifconfig $IFACE down

$ lsb_release -rd
Description: Ubuntu 11.10
Release: 11.10

$ apt-cache policy bridge-utils
bridge-utils:
  Installed: 1.5-2ubuntu1
  Candidate: 1.5-2ubuntu1
  Version table:
 *** 1.5-2ubuntu1 0
        500 http://us.archive.ubuntu.com/ubuntu/ oneiric/main i386 Packages
        100 /var/lib/dpkg/status

I was expecting to have network after boot.

I don't have network instead.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: bridge-utils 1.5-2ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-14.23-generic 3.0.9
Uname: Linux 3.0.0-14-generic i686
NonfreeKernelModules: nvidia
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Thu Jan 19 22:45:21 2012
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
SourcePackage: bridge-utils
UpgradeStatus: Upgraded to oneiric on 2011-10-16 (96 days ago)

[Steve Langasek (vorlon)]

The linked post shows "bridge_ports all" in use, which I don't believe is implemented by the bridge-utils udev rule. Stéphane, can you take a look at this?

[Stéphane Graber (stgraber)]

Actually, the udev hook uses bridge_parse_ports on bridge-ports and that function is supposed to expand the list based on regexp (with all being equivalent eth.*), so it should actually work.

I'll do some tests hopefully later today.

[Daniel Senderowicz (daniel-synchrodesign)]

I also tried with 'bridge_ports eth0' and it didn't make any difference,
the problem persists.

On Fri, 2012-01-20 at 22:47 +0000, Stéphane Graber wrote:
> ** Summary changed:
>
> - After I installed bridge-utils I don't have Network on reboot
> + bridge_ports all doesn't work at boot time
>

[Stéphane Graber (stgraber)]

I can't seem to reproduce your issue.
The worst I could get is my system hanging for 60s because one of the network scripts exitted with a non-zero return code (as the bridge was created by udev), the end result is fine though.

I'll fix the bridge script to just continue when the script already exists, that issue should be limited to 12.04 though as before that ifupdown would ignore errors from interface scripts.

[Daniel Senderowicz (daniel-synchrodesign)]

If you tell me what key configuration files are there that could effect
the difference between your and my system, I could e-mail them to you.
But this is definitively an issue.

Thanks,

Daniel

On Fri, 2012-01-20 at 23:30 +0000, Stéphane Graber wrote:

> I can't seem to reproduce your issue.
> The worst I could get is my system hanging for 60s because one of the network scripts exitted with a non-zero return code (as the bridge was created by udev), the end result is fine though.
>
> I'll fix the bridge script to just continue when the script already
> exists, that issue should be limited to 12.04 though as before that
> ifupdown would ignore errors from interface scripts.
>

[Stéphane Graber (stgraber)]

Based on your forum post, one problem is your use of resolvconf which is far from good and well working in Ubuntu 11.10, it's been fixed in 12.04.

You may want to completely remove resolvconf from your system and see if that helps.
If you still don't have network after removing resolvconf and rebooting, I'd be interesting in the output of "sudo ifdown -a && sudo ifup -a".

[Daniel Senderowicz (daniel-synchrodesign)]

I did remove completely resolvconf, including the configuration files
and the problem persists. The output of "sudo ifdown -a && sudo ifup
-a" is:

Waiting for br0 to get ready (MAXWAIT is 32 seconds).
ssh stop/waiting
ssh start/running, process 3014

and the network comes up. One more piece of information, before running
this command, I'm able to connect to machines on my LAN (e.g.
192.168.xxx.xxx) but not outside.

Thanks,

Daniel

On Sat, 2012-01-21 at 01:28 +0000, Stéphane Graber wrote:

> Based on your forum post, one problem is your use of resolvconf which is
> far from good and well working in Ubuntu 11.10, it's been fixed in
> 12.04.
>
> You may want to completely remove resolvconf from your system and see if that helps.
> If you still don't have network after removing resolvconf and rebooting, I'd be interesting in the output of "sudo ifdown -a && sudo ifup -a".
>

[Stéphane Graber (stgraber)]

An SRU has been uploaded to Ubuntu 11.10 fixing some timing issues at boot time.

Can you make sure your system is fully up to date including bridge-utils at version 1.5-2ubuntu1.1?

If the problem still happens after that, the following would probably be useful:
 - "ls -l /run/network/"
 - "initctl list"
 - /var/log/kernel.log
 - /var/log/syslog
 - ip link show
 - ip addr show
 - brctl show

Thanks

[Daniel Senderowicz (daniel-synchrodesign)]

Hi Stephane,

I updated bridge-utils and rebooted. The problem still remains. I had to
restart /etc/init.d/networking and then it worked. I am attaching the
information you requested. Thank you very much.

Regards,

Daniel

On Wed, 2012-02-22 at 16:41 +0000, Stéphane Graber wrote:

> An SRU has been uploaded to Ubuntu 11.10 fixing some timing issues at
> boot time.
>
> Can you make sure your system is fully up to date including bridge-utils
> at version 1.5-2ubuntu1.1?
>
> If the problem still happens after that, the following would probably be useful:
> - "ls -l /run/network/"
> - "initctl list"
> - /var/log/kernel.log
> - /var/log/syslog
> - ip link show
> - ip addr show
> - brctl show
>
> Thanks
>

[Chistophe S (cst)]

We had similar network problem (and slow boot) with a virtual bridge. It appeared the culprit was a somehow misonfigured ebtables package. The following command solved it:

 $ dpkg-reconfigure ebtables

debian stable amd64
bridge-utils 1.4-5
ebtables 2.0.9.2-2

[Stéphane Graber (stgraber)]

I'm starting to run out of idea here...
I just tried with the following configuration Ubuntu 12.04:
 - 6 network cards
 - 3 configured as you do in your /etc/network/interfaces
 - 3 not mentioned at all in /etc/network/interfaces
 - br0 with static IP and bridge-ports all

Can you try your configuration with Ubuntu 12.04 beta1 (or later, running updates) and ideally in a VM so we can ensure we have the exact same setup.

I had a very quick look at ebtables, at least in 12.04 it doesn't seem to contain a postinst, config or a debconf templates, so dpkg-reconfigure should be a no-op here, however it does contain an init script that could impact the system.

Result is:
---
root@stgraber-vm:~# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.52540017bb0d no eth0
       eth1
       eth2
       eth3
       eth4
       eth5
root@stgraber-vm:~# ifconfig
br0 Link encap:Ethernet HWaddr 52:54:00:17:bb:0d
          inet addr:172.23.0.50 Bcast:0.0.0.0 Mask:255.255.255.0
          inet6 addr: fe80::5054:ff:fe17:bb0d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:21859 errors:0 dropped:0 overruns:0 frame:0
          TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1086706 (1.0 MB) TX bytes:8477 (8.4 KB)

eth0 Link encap:Ethernet HWaddr 52:54:00:e2:bf:bc
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:4618 errors:0 dropped:73795 overruns:0 frame:0
          TX packets:17293 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:269714 (269.7 KB) TX bytes:1099997 (1.0 MB)
          Interrupt:11

eth1 Link encap:Ethernet HWaddr 52:54:00:88:c5:53
          inet6 addr: fe80::5054:ff:fe88:c553/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:4579 errors:0 dropped:73787 overruns:0 frame:0
          TX packets:17356 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:259356 (259.3 KB) TX bytes:1120260 (1.1 MB)
          Interrupt:11 Base address:0x4000

eth2 Link encap:Ethernet HWaddr 52:54:00:58:86:ae
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:3445 errors:0 dropped:73727 overruns:0 frame:0
          TX packets:18471 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:196380 (196.3 KB) TX bytes:1184883 (1.1 MB)
          Interrupt:11 Base address:0x6000

eth3 Link encap:Ethernet HWaddr 52:54:00:7e:91:50
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3093 errors:0 dropped:72262 overruns:0 frame:0
          TX packets:18835 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:178774 (178.7 KB) TX bytes:1207209 (1.2 MB)
          Interrupt:11 Base address:0x8000

eth4 Link encap:Ethernet HWaddr 52:54:00:19:a8:ad
          inet6 addr: fe80::5054:ff:fe19:a8ad/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Met...

[George F (cyberhawk001)]

Not sure if this is the perfect solution, but I had the exact same problem. But I got my bridged network to work as follows:

Before hand I did:
- Installed a fresh copy of Ubuntu 11.10
- Installed the latest version of bridge-utils (1.5-2ubuntu1.1)
- modified /etc/network/interfaces with bridged network settings
- Rebooted but could not get a network connection.

Than I just
- Removed bridge-utils 1.5-2ubuntu1.1
- reset /etc/network/interfaces back to its default
- Manually installed bridge-utils 1.5-2ubuntu1 from the website UbuntuUpdates.org
- recreated the bridge network settings in /etc/network/interfaces
- Rebooted and this time it worked perfectly and I could go on the net.

So long story short bridge-utils 1.5-2ubuntu1.1 has a bug that bridge-utils 1.5-2ubuntu1 does not, BUT I am sure everyone knows that already.... :)

[Steve Langasek (vorlon)]

> So long story short bridge-utils 1.5-2ubuntu1.1 has a bug that bridge-
> utils 1.5-2ubuntu1 does not, BUT I am sure everyone knows that
> already.... :)

No, that's not at all clear. This bug report was originally filed against 1.5-2ubuntu1, and the submitter reports there is no difference in behavior between the two versions. You should probably file a separate bug report for your issue.

> - Manually installed bridge-utils 1.5-2ubuntu1 from the website
> UbuntuUpdates.org

Manually downloading packages from the web and installing them is strongly contraindicated. You are completely bypassing the established security mechanisms of signed packages and opening yourself up to a man-in-the-middle attack. The appropriate way to do this is by running:
  sudo apt-get install bridge-utils/oneiric
to pull the version of the package from oneiric instead of from oneiric-updates.

[George F (cyberhawk001)]

> No, that's not at all clear. This bug report was originally filed against 1.5-2ubuntu1, and the submitter reports there is no
> difference in behavior between the two versions. You should probably file a separate bug report for your issue.

Well, i can't explain it but after i installed bridge-utils 1.5-2ubuntu1 and rebooted it worked fine for a while. The next day I turned the computer back on and i could not get it to work again. SO, there is still something else going on with it. I was so hoping it was fixed as well without it I can't even more forward with installing and setting up Xen.

So far this has been a major hurdle and really annoying to say the least.

> Manually downloading packages from the web and installing them is strongly contraindicated. You are completely bypassing
> the established security mechanisms of signed packages and opening yourself up to a man-in-the-middle attack. The
> appropriate way to do this is by running:
> sudo apt-get install bridge-utils/oneiric
>to pull the version of the package from oneiric instead of from oneiric-updates.

YA, i know it is not advised to install anything manual, BUT this is a testing server setup SO i really don't care about any security or anything. I just want bridge-utils to work SO i don't have to continue to waste hours trying to find a way to make it work.... :). If i happen to find a solution THAN, YA of course i would go back to do things the proper way, but for now, it is just simply frustrating that you can't continue cause of some bug.

In Debian there is NO such issue at all, and it and UBUNTU are so very similar that well never figured one gets stopped in your tracks so easily with Ubuntu....

Just out of shear trying something, i also installed bridge-utils 1.5-2ubuntu6, BUT still can't get it to work.

[George F (cyberhawk001)]

Actually probably never mind with the above. I installed the two packages vlan and ifenslave and it seems to work now.

[Daniel Senderowicz (daniel-synchrodesign)]

I just upgraded to 12.04. As the machine boots, it seems to connect to
the network since I can ping to IPs, however it cannot resolve DNS. In
other words if I do "ping google.com" it would say "unknown host
google.com". Please let me know what kind of specific information you
may need.

Thanks,

Daniel

On Wed, 2012-03-07 at 18:32 +0000, Stéphane Graber wrote:

> I'm starting to run out of idea here...
> I just tried with the following configuration Ubuntu 12.04:
> - 6 network cards
> - 3 configured as you do in your /etc/network/interfaces
> - 3 not mentioned at all in /etc/network/interfaces
> - br0 with static IP and bridge-ports all
>
> Can you try your configuration with Ubuntu 12.04 beta1 (or later,
> running updates) and ideally in a VM so we can ensure we have the exact
> same setup.
>
> I had a very quick look at ebtables, at least in 12.04 it doesn't seem
> to contain a postinst, config or a debconf templates, so dpkg-
> reconfigure should be a no-op here, however it does contain an init
> script that could impact the system.
>
> Result is:
> ---
> root@stgraber-vm:~# brctl show
> bridge name bridge id STP enabled interfaces
> br0 8000.52540017bb0d no eth0
> eth1
> eth2
> eth3
> eth4
> eth5
> root@stgraber-vm:~# ifconfig
> br0 Link encap:Ethernet HWaddr 52:54:00:17:bb:0d
> inet addr:172.23.0.50 Bcast:0.0.0.0 Mask:255.255.255.0
> inet6 addr: fe80::5054:ff:fe17:bb0d/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:21859 errors:0 dropped:0 overruns:0 frame:0
> TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:1086706 (1.0 MB) TX bytes:8477 (8.4 KB)
>
> eth0 Link encap:Ethernet HWaddr 52:54:00:e2:bf:bc
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:4618 errors:0 dropped:73795 overruns:0 frame:0
> TX packets:17293 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:269714 (269.7 KB) TX bytes:1099997 (1.0 MB)
> Interrupt:11
>
> eth1 Link encap:Ethernet HWaddr 52:54:00:88:c5:53
> inet6 addr: fe80::5054:ff:fe88:c553/64 Scope:Link
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:4579 errors:0 dropped:73787 overruns:0 frame:0
> TX packets:17356 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:259356 (259.3 KB) TX bytes:1120260 (1.1 MB)
> Interrupt:11 Base address:0x4000
>
> eth2 Link encap:Ethernet HWaddr 52:54:00:58:86:ae
> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
> RX packets:3445 errors:0 dropped:73727 overruns:0 frame:0
> TX packets:18471 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:196380 (196.3 KB) TX bytes:1184883 (1.1 MB)
> Interrupt:11 Base address:0x6000
>
> eth3 Link encap:Ethernet HWaddr 52:54:00:7e:91:50
> UP BROADCAST RUNN...

[Steve Langasek (vorlon)]

That seems far removed from the original issue. What are the contents of /etc/resolv.conf on your system? Where are your DNS servers being configured? The /etc/network/interfaces from the original bug report shows no dynamic or static sources of DNS, so installing resolvconf (which is required, and automatically pulled in on upgrade to 12.04) should have made /etc/resolvconf/resolv.conf.d/tail a symlink to /etc/resolvconf/resolv.conf.d/original and pulled this DNS information in when generating /etc/resolv.conf.

[Stéphane Graber (stgraber)]

Could you retry on Ubuntu 12.10 or 12.04 with all updates applied?

A few months ago I spotted a race going on with udev which got fixed and pushed back to 12.04, would be interesting to know if that helped.