linux bridge assigns mac address to the wrong port

[Adrian Pascalau]

* High level description:
linux bridge assigns mac address to the physical external interface instead of the tap interface, therefore the VM instance that uses the tap interface is not able to communicate over IP. The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces, and everything starts working.

* Pre-conditions:
I have an openstack pike running in latest centos 7 release (7.4.1708). Neutron was manually installed as described in the neutron installation guide at https://docs.openstack.org/neutron/latest/install/install-rdo.html. I have configured neutron for Network Option 2 (self service networks), however the setup I am testing here is an external flat provider network with a single cirros VM instance attached directly to it (without any router in between). The openstack environment is made of two nodes: a controller and a compute. The neutron package versions is 11.0.2-2.el7 (latest in centos 7), the bridge-utils version is 1.5-9.el7 and the kernel version is 3.10.0-693.11.1.el7.x86_64. I have tested this with cirros image cirros-0.4.0-x86_64-disk.img and cirros-0.3.5-x86_64-disk.img.

# rpm -qa | grep neutron-linuxbridge
openstack-neutron-linuxbridge-11.0.2-2.el7.noarch
# rpm -qf /usr/sbin/brctl
bridge-utils-1.5-9.el7.x86_64
# uname -a
Linux compute1 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Bridging is configured like below in both controller and compute:

ml2_conf.ini:
[ml2_type_flat]
flat_networks = physnet1

linuxbridge_agent.ini:
[linux_bridge]
physical_interface_mappings = physnet1:bond2

* Step-by-step reproduction steps:
This is how I created the provider network:
openstack network create \
  --share \
  --external \
  --provider-physical-network physnet1 \
  --provider-network-type flat \
  ExtNet1

This is how I create the provider subnet:
openstack subnet create \
  --network ExtNet1 \
  --allocation-pool start=10.20.21.96,end=$10.20.21.127 \
  --dns-nameserver 10.20.21.1 \
  --gateway 10.20.21.1 \
  --subnet-range 10.20.21.0/24 \
  ExtSubnet1

This is how I launch a cirros instance and attach it to the provider network:
openstack server create \
  --flavor m1.nano \
  --image cirros-0.4.0-x86_64-disk.img \
  --nic net-id=$(openstack network list | grep ExtNet1 | cut -d\ -f 2) \
  --security-group default \
  --key-name controller-key \
  cirros1

Based on the above, neutron creates in my compute node the following bridge:

# brctl show
bridge name bridge id STP enabled interfaces
brq75a55ef7-4a 8000.fc15b413e6a3 no bond2
                                                        tap44bc34bb-e2

bond2 is the physical interface used for the flat provider network (in access mode, no vlans) and tap44bc34bb-e2 is the tap interface attached to my cirros VM instance.

In the bridge, the bond2 is port 2, and the tap tap44bc34bb-e2 interface is port 1, and both are in forwarding mode.

# brctl showstp brq75a55ef7-4a
brq75a55ef7-4a
 <...>
 ageing time 300.00
 <...>
bond2 (2)
 port id 8002 state forwarding
  <...>
tap44bc34bb-e2 (1)
 port id 8001 state forwarding
  <...>

The network flow is like below:

default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM]

eth0 mac address is fa:16:3e:cc:dc:ec.

After the cirros VM comes up, it is not able to get an IP address from the DHCP agent, and there is no IP communication. Therefore I have to use the console and manually assign the corresponding IP address to the cirros VM eth0 interface, but still the IP connectivity does not work, since I cannot ping any external IPs, even in the same 10.20.21.0/24 subnet.

What I have found is that in the bridge forwarding table, the bridge wrongly assigns the eth0 mac address to the port 2, which is bond2 interface, instead of assigning it to the port 1, which is the tap interface. This happens only if the arp table in the cirros VM instance does not contain the mac address of the destination IP I am pinging (default gw in this case), so the cirros VM sends an arp request (Request who-has 10.203.219.1 tell 10.203.219.114). See below the eth0 mac address wrongly assigned in the forwarding table to the port 2:

# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
  2 fa:16:3e:cc:dc:ec no 0.39

However, since the eth0 mac address is wrongly assigned to the port2, the arp reply back (Reply 10.203.219.1 is-at 00:17:08:c4:52:80) does not reach anymore eth0. Using a tcpdump, I can see in the compute node the arp request going through the tap interface an the bridge, however the arp reply back does not show up anymore on the tap interface. Since the arp reply back does not reach the eth0, the arp table in the cirros VM does not contain the proper entries, and therefore the IP communication (ping in my case) does not work.

The strange thing is that after a while, for apparently no reason, a single arp reply back packet gets through the bridge and the tap interface, and the arp table gets updated with correct mac address in the cirros VM instance, and I am able to ping that IP address.

If I manually add the mac address of the destination IP I am pining into the cirros VM instance arp table, and there is no arp request sent, just icmp packets going out, then the bridge correctly assigns the eth0 mac address to the port 1, which is the tap interface, and everything starts working fine. See below the eth0 mac address correctly assigned in the forwarding table to the port 1:

# brctl showmacs brq75a55ef7-4a | grep fa:16:3e:cc:dc:ec
  1 fa:16:3e:cc:dc:ec no 0.09

The only workaround I have found to this issue is to configure the bridge ageing time to zero, converting the bridge into a hub so that it floods all packets on all ports.

It is also strange that in the bridge forwarding table, the bond0 and tap interface mac addresses are shown twice, as seen below:

# brctl showmacs brq75a55ef7-4a | grep yes
  2 fc:15:b4:13:e6:a3 yes 0.00
  2 fc:15:b4:13:e6:a3 yes 0.00
  1 fe:16:3e:cc:dc:ec yes 0.00
  1 fe:16:3e:cc:dc:ec yes 0.00

I have tested this in an all-in-one openstack installation running in a vmware centos 7 VM, that has a single interface configured in promiscuous mode in the vmware portgroup (therefore no interface bonding like in the previous setup), and the same issue happens, therefore I do not think this is caused by the bonding interface.

[Lujin Luo (luo-lujin)]

I am trying to reproduce this locally. Please kindly wait for a while. If others are willing to reproduce, you are much more than welcome.

[Adrian Pascalau (adrian-pascalau)]

I have investigated this further, and it looks like the issue is in my external network, since a device is bouncing back the arp request, and this is why the bridge assigns it to the bond2 interface. So what happens is the following:

default gw<-->physical switch<-->[bond2 bridge tap]<-->[eth0 cirrosVM]

The arp request goes out on the eth0 interface, and enters the bridge on the tap interface. The bridge assigns the eth0 mac address to the tap interface, and sends the arp request out on the bond2 interface. Now some deice on the left side of the bridge (either the physical switch or the default gw), broadcasts that arp requests back, therefore the same arp request enters back the bridge on the bond2 interface, and the bridge assigns the source mac address of the arp request (which is still the eth0 mac address) to the bond2 port in the forwarding table, which causes the behavior I have noticed...

This also explains why I see 2 arp requests and a single arp reply when tracing:

# tcpdump -n -i bond2 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond2, link-type EN10MB (Ethernet), capture size 262144 bytes
06:13:48.581758 ARP, Request who-has 10.20.21.1 tell 10.20.21.114, length 28
06:13:48.581791 ARP, Request who-has 10.20.21.1 tell 10.20.21.114, length 28
06:13:48.582221 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46

I am really sorry for all the trouble.

[Radu Pantiru (radupantiru)]

Had the same problem when building a new VM and could not get DHCP to work as the reply traffic got lost when the bridge sent it to the wrong port.

The solution was to set dhcp_broadcast_reply = true in /etc/neutron/dhcp_agent.ini