Multicast traffic not propating correctly over linux bridge

[James Page]

There's a lot of supposition in the title of this bug but its currently my best guess.

In this deployment, I have a number of services running in LXC containers across multiple physical hosts; each service is clustered across three units, all on separate physical hosts, using corosync and pacemaker; when using multicast to support cluster communication, I occasionally see a container drop out of the cluster and use its isolation response (to shutdown all managed services); when using unicast I've not yet see this same problem.

LXC containers are bridged to the main physical network using a linux bridge:

  eth0 <-> juju-br0 <-> vethXXX <-> | vethXXX |

All MTU's are standard (1500).

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: lxc 1.0.6-0ubuntu0.1
ProcVersionSignature: User Name 3.13.0-40.69-generic 3.13.11.10
Uname: Linux 3.13.0-40-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
Date: Mon Dec 15 17:20:08 2014
ProcEnviron:
 TERM=screen-bce
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Dec 17 10:16 seq
 crw-rw---- 1 root audio 116, 33 Dec 17 10:16 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 14.04
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Dell Inc. PowerEdge R610
Package: lxc 1.0.6-0ubuntu0.1
PackageArchitecture: amd64
PciMultimedia:

ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-3.13.0-43-generic root=UUID=5a86874d-8bbd-4e7a-b73e-17c914de390b ro
ProcEnviron:
 TERM=screen-bce
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-43-generic root=UUID=5a86874d-8bbd-4e7a-b73e-17c914de390b ro
ProcVersionSignature: User Name 3.13.0-43.72-generic 3.13.11.11
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty uec-images trusty uec-images apparmor
Uname: Linux 3.13.0-43-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy libvirtd netdev plugdev sudo video
_MarkForUpload: True
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx
dmi.bios.date: 08/18/2011
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 6.0.7
dmi.board.name: 0F0XJ6
dmi.board.vendor: Dell Inc.
dmi.board.version: A11
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr6.0.7:bd08/18/2011:svnDellInc.:pnPowerEdgeR610:pvr:rvnDellInc.:rn0F0XJ6:rvrA11:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R610
dmi.sys.vendor: Dell Inc.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Page (james-page)]

I've done a bit more testing, and here are my observations:

1) rebooting a container internally/stop-start externally using lxc-*

Works OK - container re-joins the corosync cluster just fine.

2) rebooting the physical host

Fails - containers never re-join the corosync cluster and perform the configured isolation response which is to stop everything.

That said, if I reconfigure the cluster to using a new multicast address, the cluster reforms OK.

So it would appear that cross server multicast is not being restored on a physical server reboot; I guess this could be something todo with switch configuration as well.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1402763

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[James Page (james-page)]

Things appear to be somewhat random; however setting the multicast_querier flag to 1 resulted in all my clustered spring back to life:

for i in `seq 0 10`; do juju ssh $i "echo -n 1 | sudo tee /sys/devices/virtual/net/juju-br0/bridge/multicast_querier"; done

Juju creates a bridge where this is not enabled by default.

[James Page (james-page)]

For context:

http://en.wikipedia.org/wiki/IGMP_snooping#IGMP_querier

[James Page (james-page)]

I enabled the multicast_querier using a udev rule for juju-br0; however physical host reboots are still resulting in impact lxc containers not re-joining the cluster; toggling the querier off/on again resolves the issue so I'm guessing some sort of race.

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

apport information

[James Page (james-page)]

Even post toggling the querier off/no, multicast is still unreliable, with cluster members failing to transmit data successfully to each other.

[James Page (james-page)]

Just to fill in a bit more detail, the physical hosts have multiple lxc containers (~3 each), any of which will be participating in a different multicast group; containers in the same group are spread across different physical hosts.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Stéphane Graber (stgraber)]

Anything else that's special on that network, e.g. non-standard MTU?

[James Page (james-page)]

Stephane

The network itself is pretty stock; the mtu on the switches is set to 9000; other than that its pretty much the standard cisco configuration for the switch model.

I have the full details (but can't put them here :-)).

[guigui (gfraysse-free)]

Have the same issue not juju related at all : created a single LXC container on ubuntu precise :
Here are the value for multicast querier
cat /sys/devices/virtual/net/lxcbr0/bridge/multicast_querier
1

From container : ping 239.255.255.250 (UPnP adress) does not work but works fine on host. Otherwise container network is fine.