bonding inside a bridge does not update ARP correctly when bridged net accessed from within a VM
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Medium
|
Unassigned | ||
qemu-kvm (Ubuntu) |
Invalid
|
Medium
|
Serge Hallyn |
Bug Description
Binary package hint: qemu-kvm
Description: Ubuntu 10.4.2
Release: 10.04
When setting a KVM host with a bond0 interface made of eth0 and eth1 and using this bond0 interface for a bridge to KVM VMs, the ARP tables do not get updated correctly so it is not possible for a VM to reach an IP on the bridged network up until that remote system has pinged the VM itself.
Reproducible: 100%, with any of the load balancing mode
How to reproduce the problem
- Prerequisites:
1 One KVM system with 10.04.02 server, configured as a virtual host is needed. The following /etc/network/
# grep -v ^# /etc/network/
auto lo
iface lo inet loopback
auto bond0
iface bond0 inet manual
post-up ifconfig $IFACE up
pre-down ifconfig $IFACE down
bond-slaves none
bond_mode balance-rr
bond-downdelay 250
bond-updelay 120
auto eth0
allow-bond0 eth0
iface eth0 inet manual
bond-master bond0
auto eth1
allow-bond0 eth1
iface eth1 inet manual
bond-master bond0
auto br0
iface br0 inet dhcp
# dns-* options are implemented by the resolvconf package, if installed
bridge-ports bond0
bridge-stp off
bridge-fd 9
bridge-hello 2
bridge-maxage 12
bridge_max_wait 0
One VM running Maverick 10.10 server, standard installation, using the following /etc/network/
grep -v ^# /etc/network/
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 10.153.107.92
netmask 255.255.255.0
network 10.153.107.0
broadcast 10.153.107.255
--------------
On a remote bridged network system :
$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
On KVMhost
$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0
On VM
$ arp -an
? (10.153.107.61) at <incomplete> on eth0
1) Test #1 : ping from VM (10.153.107.92) to remote bridged network system (10.153.107.80) :
- On remote bridged network system :
caribou@marvin:~$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
- On KVMhost
ubuntu@VMhost:~$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0
- On VM
ubuntu@vm1:~$ ping 10.153.107.80
PING 10.153.107.80 (10.153.107.80) 56(84) bytes of data.
From 10.153.107.92 icmp_seq=1 Destination Host Unreachable
From 10.153.107.92 icmp_seq=2 Destination Host Unreachable
From 10.153.107.92 icmp_seq=3 Destination Host Unreachable
^C
--- 10.153.107.80 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3010ms
pipe 3
ubuntu@vm1:~$ arp -an
? (10.153.107.61) at <incomplete> on eth0
? (10.153.107.80) at <incomplete> on eth0
2) Test #2 : ping from remote bridged network system (10.153.107.80) to VM (10.153.107.92) :
- On remote bridged network system :
$ ping 10.153.107.92
PING 10.153.107.92 (10.153.107.92) 56(84) bytes of data.
64 bytes from 10.153.107.92: icmp_req=1 ttl=64 time=327 ms
64 bytes from 10.153.107.92: icmp_req=2 ttl=64 time=158 ms
64 bytes from 10.153.107.92: icmp_req=3 ttl=64 time=157 ms
^C
--- 10.153.107.92 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 157.289/
caribou@marvin:~$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
? (10.153.107.92) à 52:54:00:8c:e0:3c [ether] sur tap0
- On KVMhost
$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0
- On VM
arp -an
? (10.153.107.61) at <incomplete> on eth0
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on eth0
3) Test #3 : New ping from VM (10.153.107.92) to remote bridged network system (10.153.107.80) :
- On remote bridged network system :
$ arp -an
? (10.153.107.188) à 00:1c:c4:6a:c0:dc [ether] sur tap0
? (16.1.1.1) à 00:17:33:e9:ee:3c [ether] sur wlan0
? (10.153.107.52) à 00:1c:c4:6a:c0:de [ether] sur tap0
? (10.153.107.92) à 52:54:00:8c:e0:3c [ether] sur tap0
- On KVMhost
ubuntu@VMhost:~$ arp -an
? (10.153.107.80) at ee:99:73:68:f0:a5 [ether] on br0
- On VM
ubuntu@vm1:~$ ping 10.153.107.80
PING 10.153.107.80 (10.153.107.80) 56(84) bytes of data.
64 bytes from 10.153.107.80: icmp_req=1 ttl=64 time=154 ms
64 bytes from 10.153.107.80: icmp_req=2 ttl=64 time=170 ms
64 bytes from 10.153.107.80: icmp_req=3 ttl=64 time=154 ms
^C
--- 10.153.107.80 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 154.072/
tcpdump traces are available for those tests. Test system is available upon request.
Workaround:
Use the bonded device in "active-backup" mode
ProblemType: Bug
DistroRelease: Ubuntu 10.04.02
Package: qemu-kvm-
Uname: Linux 2.6.35-25-serverr x86_64
Architecture: amd64
description: | updated |
Changed in qemu-kvm (Ubuntu): | |
importance: | Undecided → Medium |
Changed in qemu-kvm (Ubuntu): | |
status: | Incomplete → Confirmed |
Changed in qemu-kvm (Ubuntu): | |
assignee: | nobody → Serge Hallyn (serge-hallyn) |
no longer affects: | qemu |
Thanks for reporting this bug and the detailed reproduction instructions. I would mark it high, but since you offer a workaround I'll mark it medium instead.
What does your /etc/modprobe. d/bonding show?
I've not used this combination myself, but from those who have, a few things do appear fragile, namely:
1. if you are using 802.3ad, you need trunking enabled on the physical switch
2. some people find that turning stp on helps (http:// www.linuxquesti ons.org/ questions/ linux-networkin g-3/bridging- a-bond- 802-3ad- only-works- when-stp- is-enabled- 741640/)
But I'm actually wondering whether this patch:
http:// permalink. gmane.org/ gmane.linux. network/ 159403
may be needed. If so, then even the natty kernel does not yet have that fix.
I am marking this as affecting the kernel, since I believe that is where the bug lies.