bridges cannot have a mtu > 1500 by themselves

[Simon Déziel]

A linux bridge always adopts the smallest MTU of the enslaved devices. When no device are enslaved, it defaults to a MTU of 1500 and refuses to use a larger one. This is problematic when using bridges enslaving only virtual NICs (vnetX) like it's common with KVM guests.

Steps to reproduce the problem

1) sudo ip link add br-test0 type bridge # create an empty bridge
2) sudo ip link set br-test0 mtu 9000 # attempt to set MTU > 1500
3) ip link show dev br-test0 # confirm MTU

Here, 2) returns "RTNETLINK answers: Invalid argument". One (cumbersome) way around this is:

4) sudo modprobe dummy
5) sudo ip link set dummy0 mtu 9000 master br-test0

Then the bridge's MTU can be changed from anywhere to 9000.

This bug is especially annoying for the virtualization case because the KVM's tap driver will by default adopt the bridge's MTU on startup making it impossible (without the workaround) to use a large MTU on the guest VMs.

Additional information:

$ lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04
$ apt-cache policy linux-image-3.13.0-41-generic iproute2
linux-image-3.13.0-41-generic:
  Installed: 3.13.0-41.70
  Candidate: 3.13.0-41.70
  Version table:
 *** 3.13.0-41.70 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
iproute2:
  Installed: 3.12.0-2
  Candidate: 3.12.0-2
  Version table:
 *** 3.12.0-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-41-generic 3.13.0-41.70
ProcVersionSignature: Ubuntu 3.13.0-41.70-generic 3.13.11.11
Uname: Linux 3.13.0-41-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: simon 4594 F.... pulseaudio
CurrentDesktop: Unity
CurrentDmesg: dmesg: klogctl failed: Operation not permitted
Date: Wed Dec 3 23:02:25 2014
HibernationDevice: RESUME=UUID=8e7a3220-158f-413b-81b2-55b236bb1f3f
InstallationDate: Installed on 2014-01-26 (311 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140124)
MachineType: LENOVO 2516CTO
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-41-generic root=/dev/mapper/crypt-root ro quiet splash cryptopts=target=crypt,source=/dev/sda1,lvm=crypt-root possible_cpus=4 nmi_watchdog=0 vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-41-generic N/A
 linux-backports-modules-3.13.0-41-generic N/A
 linux-firmware 1.127.10
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/14/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: 6IET85WW (1.45 )
dmi.board.name: 2516CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr6IET85WW(1.45):bd02/14/2013:svnLENOVO:pn2516CTO:pvrThinkPadT410:rvnLENOVO:rn2516CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2516CTO
dmi.product.version: ThinkPad T410
dmi.sys.vendor: LENOVO

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue occur in a previous version of Ubuntu, or is this a new issue?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.18 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.18-rc7-vivid/

[Simon Déziel (sdeziel)]

On 12/04/2014 01:44 PM, Joseph Salisbury wrote:
> Did this issue occur in a previous version of Ubuntu, or is this a new
> issue?

This is reproducible on 12.04 too.

> Would it be possible for you to test the latest upstream kernel? Refer
> to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
> v3.18 kernel[0].

Also reproducible with:

$ uname -a
Linux xeon 3.18.0-031800rc7-generic #201411302035 SMP Mon Dec 1 01:36:38
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Thanks

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Andy Whitcroft (apw)]

Looking at the current mainline code, when you set the MTU is it clamped by the bridge wide "lowest" mtu:

  static int br_change_mtu(struct net_device *dev, int new_mtu)
  {
        struct net_bridge *br = netdev_priv(dev);
        if (new_mtu < 68 || new_mtu > br_min_mtu(br))
                return -EINVAL;
  [...]
  }

Which is calculated rather simplistically where there are no attached devices:

  int br_min_mtu(const struct net_bridge *br)
  {
  [...]
        if (list_empty(&br->port_list))
                mtu = ETH_DATA_LEN;
  [...]

It is not entirely clear why we care to clamp the MTU artificially at this level, as we will reevaluate the mtu and lower it when we do add a new bridge port.

All of that said, if the MTU was set on the KVM tap before it was added then the bridge MTU should follow that device, and that appears to be the expected use model from the way the code is written.

[Simon Déziel (sdeziel)]

On 12/05/2014 01:20 AM, Andy Whitcroft wrote:
> Which is calculated rather simplistically where there are no attached
> devices:
>
> int br_min_mtu(const struct net_bridge *br)
> {
> [...]
> if (list_empty(&br->port_list))
> mtu = ETH_DATA_LEN;
> [...]
>
> It is not entirely clear why we care to clamp the MTU artificially at
> this level, as we will reevaluate the mtu and lower it when we do add a
> new bridge port.

Thanks for diving in the code Andy. IMHO, having the MTU defaulting to
1500 for an empty bridge makes sense but one should be able to manually
set it to a higher MTU.

> All of that said, if the MTU was set on the KVM tap before it was added
> then the bridge MTU should follow that device, and that appears to be
> the expected use model from the way the code is written.

Indeed, if the tap's MTU was bigger when it was enslaved it would just
work. Currently it works the other way around: the tap adopts the
bridge's MTU [1]

Regards,
Simon

[1]: https://www.redhat.com/archives/libvir-list/2008-December/msg00083.html

[Thiago Martins (martinx)]

Also affects Linux 3.19 on Ubuntu Trusty (linux-generic-lts-vivid).

It seems that the only way to create a bridge with MTU=9000 is by creating it on top of a dummy interface, with mtu 9000 already configured... Very annoying...

[Saverio (saveriolibrary)]

I am wondering if there has been any progress on this, and if a bug report has been opened to the kernel devs.
This is impacting kvm deployments, and while there are workarounds, they are all rather ugly (read: scripts that adjust tun MTUs after VMs boot).

[Tejeev Patel (tejeevpatel)]

We ran into this while training for NOC. Deploying on our laptop labs so not sure of greater implications but it's tied us up for a good bit so I figured I'd bump this thread.

so:
**** BUMP ****

[Tejeev Patel (tejeevpatel)]

If any one runs into this issue in the future, be sure you are using virtio for your network interfaces for the vm's.

[Ken Sharp (kennybobs)]

LXC displays similar behavious in Xenial. The "solution" was to set the MTUs for all the slave interfaces, and all the interfaces within the guests, then finally set the bridge MTU, otherwise it will simply refuse. Worked fine.

Haven't yet looked into the configs to see if I can do this permanently.

[engin (dumlu)]

64bit xenial, same issue here;

it's sad, not able to use jumbo frames with in lxd guests.

[Trent Lloyd (lathiat)]

This has been fixed, I'm not entirely sure when, but you can create a bridge with a specific MTU.

Additionally since v4.17 when you set the MTU, the bridge will stop automatically changing it when interfaces are added/removed:
https://github.com/torvalds/linux/commit/804b854d374e39f5f8bff9638fd274b9a9ca7d33

However until a commit I'm pushing now, that worked if you set the MTU after creation (ip link set br0 mtu 9000) but failed if the MTU was set during creation (ip link add br0 type bridge mtu 9000) - which is also the equivalent to how systemd-networkd does it. So the MTU may shrink if you accidentally add lower MTU interfaces to the bridge. But LXC/LXD/etc typically set the MTU to match the MTU the bridge currently has so mostly this should work as desired since about 2018/v4.17 at least.