Linux Bridge: can't change the VM's bridge and tap interface MTU at Compute node.

[Danny Choi]

I use DevStack to deploy OpenStack with Linux Bridge instead of OVS in a multi-node set up.

I¹m testing jumbo frames and want to set MTU to 9000.

At the Network node, the bridges and tap interfaces are created with MTU = 9000:

localadmin@qa4:~/devstack$ brctl show
bridge name bridge id STP enabled interfaces
brq09047ecb-1c 8000.7c69f62c4f2f no eth1
                                                                                                      tapedbcd5b1-a6
brq319688ab-93 8000.3234d6ee3a18 no bond0.300
                                                                                                      tap4e230a86-cb
                                                                                                      tapfddaf12e-85
virbr0 8000.000000000000 yes

localadmin@qa4:~/devstack$ ifconfig brq09047ecb-1c
brq09047ecb-1c Link encap:Ethernet HWaddr 7c:69:f6:2c:4f:2f
          inet6 addr: fe80::3c79:c8ff:fe23:2fe7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:696 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:30617 (30.6 KB) TX bytes:648 (648.0 B)

localadmin@qa4:~/devstack$ ifconfig brq319688ab-93
brq319688ab-93 Link encap:Ethernet HWaddr 32:34:d6:ee:3a:18
          inet6 addr: fe80::e0ec:3bff:fe09:4318/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:30 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4236 (4.2 KB) TX bytes:648 (648.0 B)

localadmin@qa4:~/devstack$ ifconfig tapedbcd5b1-a6
tapedbcd5b1-a6 Link encap:Ethernet HWaddr ae:fb:64:53:f7:2d
          inet6 addr: fe80::acfb:64ff:fe53:f72d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:65 errors:0 dropped:0 overruns:0 frame:0
          TX packets:947 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:10223 (10.2 KB) TX bytes:80510 (80.5 KB)

localadmin@qa4:~/devstack$ ifconfig tap4e230a86-cb
tap4e230a86-cb Link encap:Ethernet HWaddr 32:34:d6:ee:3a:18
          inet6 addr: fe80::3034:d6ff:feee:3a18/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:2073 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2229 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8878139 (8.8 MB) TX bytes:8914532 (8.9 MB)

localadmin@qa4:~/devstack$ ifconfig tapfddaf12e-85
tapfddaf12e-85 Link encap:Ethernet HWaddr d2:33:29:9b:2c:e8
          inet6 addr: fe80::d033:29ff:fe9b:2ce8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:152 errors:0 dropped:0 overruns:0 frame:0
          TX packets:295 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15237 (15.2 KB) TX bytes:51849 (51.8 KB)

The instance launched at the Compute node has interface eth0 MTU = 9000:

ubuntu@qa5-vm2:~$ ifconfig
eth0 Link encap:Ethernet HWaddr fa:16:3e:05:36:58
          inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe05:3658/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:1169 errors:0 dropped:0 overruns:0 frame:0
          TX packets:384 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1408206 (1.4 MB) TX bytes:1336535 (1.3 MB)

However, the associated bridge and tap interface MTU is set to default 1500:

localadmin@qa5:~/devstack$ brctl show
bridge name bridge id STP enabled interfaces
brq319688ab-93 8000.6805ca302558 no bond0.300
                                                                                                          tapa7acee8a-54
virbr08000.000000000000 yes

localadmin@qa5:~/devstack$ ifconfig brq319688ab-93
brq319688ab-93 Link encap:Ethernet HWaddr 68:05:ca:30:25:58
          inet6 addr: fe80::a8ef:dfff:feb1:8eec/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:162 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:910353 (910.3 KB) TX bytes:648 (648.0 B)

localadmin@qa5:~/devstack$ ifconfig tapa7acee8a-54
tapa7acee8a-54 Link encap:Ethernet HWaddr fe:16:3e:05:36:58
          inet6 addr: fe80::fc16:3eff:fe05:3658/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:395 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1180 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1338757 (1.3 MB) TX bytes:1408932 (1.4 MB)

localadmin@qa5:~/devstack$ ifconfig bond0.300
bond0.300 Link encap:Ethernet HWaddr 68:05:ca:30:25:58
          inet6 addr: fe80::6a05:caff:fe30:2558/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:11507 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11590 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14831272 (14.8 MB) TX bytes:14943870 (14.9 MB)

As a result, jumbo frames are fragmented coming out from the Compute node.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/174558

[James Denton (james-denton)]

Patch as-is resulted in traceback:

2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1014, in daemon_loop
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 870, in process_network_devices
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added_updated(devices_added_updated)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 910, in treat_devices_added_updated
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent device_details['port_id']):
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 435, in add_interface
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 401, in add_tap_interface
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent self.ensure_tap_mtu(tap_device_name, phy_dev_name)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 425, in ensure_tap_mtu
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent ip_lib.IPDevice(tap_dev_name).link.set_mtu(phy_dev_mtu)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 258, in set_mtu
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent self._as_root('set', self.name, 'mtu', mtu_size)
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent File "/usr/local/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 242, in _as_root
2015-04-16 19:16:24.496 58732 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent kwargs.get('use_root_namespace', False))
2015-04-16 19:16:24.496 58732 TRACE neutron....

[James Denton (james-denton)]

To demonstrate:

Before:

#ip link show br-vlan.440 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
br-vlan.440@br-vlan: mtu 1500 brq33fecdff-b3

#ip link show tap346c55b7-97 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap346c55b7-97: mtu 1500 brq33fecdff-b3

#ip link show tap59ddc0ac-7c | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap59ddc0ac-7c: mtu 1500 brq33fecdff-b3

Change physical MTU & Restart Agent:

# ip link set mtu 9000 dev br-vlan.440
# service neutron-linuxbridge-agent restart

After:

#ip link show br-vlan.440 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
br-vlan.440@br-vlan: mtu 9000 brq33fecdff-b3

#ip link show tap346c55b7-97 | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap346c55b7-97: mtu 9000 brq33fecdff-b3

#ip link show tap59ddc0ac-7c | grep mtu | awk {'print $2 " " $4 " " $5 " " $9'}
tap59ddc0ac-7c: mtu 9000 brq33fecdff-b3

[Darragh O'Reilly (darragh-oreilly)]

oops - forgot I disabled rootwrap. Will do a new patchset.

[Baodong (Robert) Li (baoli)]

Darragh, thanks for coming up the patch. We are working with Danny (the bug submitter), and actually trying to come up with a patch that would address the issue as a whole. But sort of late to assign the bug to ourself.

I looked at your patch, and felt that it doesn't seem to address the issue completely.
   -- the mtu on the vlan/vxlan interface should be set automatically per the existing config options (network_device_mtu or possibly veth_mtu or the MTU patches merged in Kilo)
   -- the bridge's MTU should be set properly as well.

Let us know if we can work together on this.

[Darragh O'Reilly (darragh-oreilly)]

Robert: okay, but it is not clear to me what is missing. Can you please test this patch and report its short-comings.

This patch relies on several things about linux bridges and devices that are not obvious:

- With Nova/libvirt/QEMU/KVM, the tap devices are created with the same MTU as the linux bridge.
- The MTU on a linux bridge is always the smallest of all the MTUs of all the devices it enslaves.
- A VLAN device automatically gets the same MTU as its parent when it is created.
- A VXLAN devices gets an MTU 50 bytes lower than its parent.

# ip -o l | cut -d' ' -f2,5 | grep eth1
eth1: 9000

# ip link add link eth1 name eth1.10 type vlan id 10
# ip -o l | cut -d' ' -f2,5 | grep eth1
eth1: 9000
eth1.10@eth1: 9000

# brctl addbr br1
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 1500

# brctl addif br1 eth1.10
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 9000

# sudo ip link add vxlan1 type vxlan id 1 group 239.0.0.1 dev eth1
# ip -o l | cut -d' ' -f2,5 | grep vxlan1
vxlan1: 8950

# brctl addif br1 vxlan1
# ip -o l | cut -d' ' -f2,5 | grep br1
br1: 8950

# brctl show
bridge name bridge id STP enabled interfaces
br1 8000.080027eb45b4 no eth1.10
       vxlan1

[Danny Choi (dannchoi)]

Hi Darragh, I'll test the patch.

I thought you are going to post a new patch set to fix the tracebacks, or should I just use patch set #1?

[Darragh O'Reilly (darragh-oreilly)]

Danny, yeah I though I needed to catch up on the recent changes to rootwrap, but maybe not. All the other places in the code do it the same way as the patch.

~$ grep -r link.set_mtu /opt/stack/neutron/ | tail -2
/opt/stack/neutron/neutron/agent/linux/interface.py: root_veth.link.set_mtu(self.conf.network_device_mtu)
/opt/stack/neutron/neutron/agent/linux/interface.py: ns_veth.link.set_mtu(self.conf.network_device_mtu)

James - what version are you using?

[James Denton (james-denton)]

Sorry - should have mentioned it was Juno.

[Darragh O'Reilly (darragh-oreilly)]

The patch does not have a rootwrap problem with trunk. There has been major changes in rootwrap since Juno.

[Danny Choi (dannchoi)]

Hi Darragh,

I'm using Kilo and did not see any traceback with your patch.

I'm using a multi-node setup. So I applied your patch to the Compute node and restarted the q-agt service.

When a VM is launched at the Compute node, the correct MTU 9000 is configured at all interfaces and devices:

1. VM eth0 interface:
$ ifconfig eth0 | grep MTU
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1

2. Compute node:
localadmin@qa5:/opt/stack/neutron$ brctl show
bridge name bridge id STP enabled interfaces
brqb5924fb1-0f 8000.6805ca30255b no eth5.300
       tap1425c7b3-f2
virbr0 8000.000000000000 yes

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep eth5
eth5: 9000
eth5.300@eth5: 9000

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep brq
brqb5924fb1-0f: 9000

localadmin@qa5:/opt/stack/neutron$ ip -o l | cut -d' ' -f2,5 | grep tap
tap1425c7b3-f2: 9000

[James Denton (james-denton)]

Fair enough. Thanks for the quick turnaround.

[Baodong (Robert) Li (baoli)]

Darragh, I realized that by setting the mtu on the physical interface, the sub interfaces will inherit from that. Thank you for pointing that out.

[Darragh O'Reilly (darragh-oreilly)]

ok good. So with this patch I think the behavior is the same as nova-network, and is what people expect. The interfaces specified in device_mappings should have their MTUs set by the operating system startup scripts - these will usually be complex devices like bonds with many other parameters.

One last thing, you will need to set network_device_mtu in l3_agent.ini so the other side of the veth (the side in the namespace) gets the correct MTU,

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174558
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6cf92011143eb55adda180ffac91886566fc7826
Submitter: Jenkins
Branch: master

commit 6cf92011143eb55adda180ffac91886566fc7826
Author: Darragh O'Reilly <email address hidden>
Date: Thu Apr 16 18:21:03 2015 +0000

    lb-agent: ensure tap mtu is the same as physical device

    On compute-nodes, Nova creates the bridge with the tap before
    the physical is in the bridge. This causes the tap to have the
    default 1500 MTU which may be different to what is on the physical.
    With this patch the linuxbridge agent ensures that the MTU on the
    tap device is the same as what is on the physical device.

    Change-Id: Id1a4f662ec33ca0333c15eb210366bc850d0d54c
    Closes-Bug: 1443607

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/207171

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207171
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4f9409d0bc2bdaab20fd067a1ec0c522b5ef2784
Submitter: Jenkins
Branch: stable/kilo

commit 4f9409d0bc2bdaab20fd067a1ec0c522b5ef2784
Author: Darragh O'Reilly <email address hidden>
Date: Thu Apr 16 18:21:03 2015 +0000

    lb-agent: ensure tap mtu is the same as physical device

    On compute-nodes, Nova creates the bridge with the tap before
    the physical is in the bridge. This causes the tap to have the
    default 1500 MTU which may be different to what is on the physical.
    With this patch the linuxbridge agent ensures that the MTU on the
    tap device is the same as what is on the physical device.

    Change-Id: Id1a4f662ec33ca0333c15eb210366bc850d0d54c
    Closes-Bug: 1443607
    (cherry picked from commit 6cf92011143eb55adda180ffac91886566fc7826)

[Xiao (shaw-leon)]

Is this done for openvswitch bridges?

[Prashant (pczanwar)]

Has anyone tested this on Juno ? I tried this today, and i dont see change happening ..