DHCP's "Option interface-mtu 9000" is being ignored on bridge interface br0

[Victor Tapia]

In an env with jumbo frames enabled, and using MAAS as DHCP server, the client receives the following IPv4 lease:

$ cat /var/lib/dhcp/dhclient.br0.leases
lease {
  interface "br0";
  fixed-address 10.230.20.26;
  filename "pxelinux.0";
  option subnet-mask 255.255.248.0;
  option dhcp-lease-time 43200;
  option routers 10.230.16.1;
  option dhcp-message-type 5;
  option dhcp-server-identifier 10.230.20.1;
  option domain-name-servers 10.230.20.1;
  option interface-mtu 9000;
  option broadcast-address 10.230.23.255;
  option domain-name "ctsstack.qa.1ss";
  renew 3 2014/12/17 16:48:15;
  rebind 3 2014/12/17 21:52:09;
  expire 3 2014/12/17 23:22:09;
}

The interfaces show the following config after boot:

$ ifconfig br0
br0 Link encap:Ethernet HWaddr a0:d3:c1:01:9d:58
          inet addr:10.230.20.26 Bcast:10.230.23.255 Mask:255.255.248.0
          inet6 addr: fe80::a2d3:c1ff:fe01:9d58/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:530530 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1591569 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:68713489 (68.7 MB) TX bytes:213710979 (213.7 MB)

$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr a0:d3:c1:01:9d:58
          inet6 addr: fe80::a2d3:c1ff:fe01:9d58/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:10539274 errors:0 dropped:3394 overruns:0 frame:454
          TX packets:2627412 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2320560616 (2.3 GB) TX bytes:3562885157 (3.5 GB)
          Interrupt:32

"option interface-mtu 9000;" from the lease file is being ignored by br0. Could it be related to eth0 MTU size? If that's the case, shouldn't both interfaces be updated?

Other info:

$ brctl show br0
bridge name bridge id STP enabled interfaces
br0 8000.a0d3c1019d58 no eth0

$ cat /etc/network/eth0.config
iface eth0 inet manual

auto br0
iface br0 inet dhcp
  bridge_ports eth0

[Robie Basak (racb)]

Presumably this is related to the DHCP client via dhclient and dhclient-script, so presumably should be a task on isc-dhcp?

I see code in dhclient-script that sets the MTU based on the DHCP option, but it doesn't have any special behaviour for br0. So I'm wondering if this code is being used at all, or if some other interaction is causing it to fail in this case.

Why is br0 being set up to bridge eth0? Is something doing this automatically or is this configuration necessary for something (if so, what?)?

Without using br0 and just using plain eth0, does the interface-mtu option work correctly?

[Robie Basak (racb)]

https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1355813/comments/3 might be relevant here. Kapil says: "...you can't really set a bridge mtu, you have to set it for all the containers on the bridge, ie. bridges take the lowest mtu of their interfaces. so instead we have to set all the containers mtu."

If so, I'm not sure it really makes sense for the DHCP client to set the MTU on eth0 when receiving DHCP over br0. I suppose it depends on the use case though.

[Victor Tapia (vtapia)]

Hi Robie,

Let me answer your questions:

> Why is br0 being set up to bridge eth0? Is something doing this automatically or is this configuration necessary for something (if so, what?)?
This configuration (br0 on a single physical interface) is automatically generated by deploying openstack Juno (I don't know what particular component does it, atm) on Trusty. In our case it's not necessary/mandatory, but just the result of the installation. I don't know what consequences we might face by disabling/removing br0, though.

> Without using br0 and just using plain eth0, does the interface-mtu option work correctly?
Will do asap, but on a first guess I think it would work

>... If so, I'm not sure it really makes sense for the DHCP client to set the MTU on eth0 when receiving DHCP over br0. I suppose it depends on the use case though.
I agree. In this particular scenario, having just one physical interface sharing its mac address with the bridge, I think it makes sense to expect the interface's MTU being set (there's no other interface acting as a constraint). Other situations, with several interfaces, manual config might be mandatory.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Curtis Hovey (sinzui)]

From James's duplicate bug:

In our MAAS based openstack deployment, we provide MTU via DHCP to all servers to enable jumbo frames.

This works fine until juju creates the bridge devices for LXC or KVM containers, at which point we reliably lose the MTU 9000 setting on the bridge, resulting in all sorts of MTU related fun in things like corosync and percona-cluster which are running under lxc.

We've fixed this by appending an explicit mtu set to the bridge definition in /etc/network/interfaces:

auto juju-br0
iface juju-br0 inet dhcp
    bridge_ports eth0
    post-up /sbin/ip link set eth0 mtu 9000

After making this change, we can reliably reboot physical hosts container lxc containers, an not break our clusters.

The issue is that MTU has to be set on the underlying device, not the bridge itself - this may actually be something we can fix in the dhcpclient tools

[Dimiter Naydenov (dimitern)]

Since 1.23 there's no need to create juju-br0 (or as it used to be called before - br0) at initial boot, so this should solve the issue.

[Gema Gomez (gema)]

This issue is still a problem in 1.23, we just reproduced it.

Also, not sure why this bug was marked as fix released when there was no fix to release, just the assumption that this problem would not happen due to some new condition that doesn't seem to be met by our environment either. I think in this case the bug should have been marked either "invalid" or "not going to fix".

[Dimiter Naydenov (dimitern)]

juju-br0 was reintroduced in 1.23.3 as the default behavior in MAAS, unless the "address-allocation" feature flag is enabled.

AIUI the juju-br0 "swallows" the "interface-mtu" DHCP setting coming from MAAS. It doesn't happen when eth0 is not bound to juju-br0. So it looks like this is a pre-existing issue since for ever.

I'm open to suggestions how Juju can work around the issue (e.g. by generating a slightly different /etc/network/interfaces file with the juju-br0 definition and/or brctl commands).

[Dimiter Naydenov (dimitern)]

Depending on the version of juju used, one option is to set:

lxc-default-mtu: 9000

in environments.yaml for that environment and re-bootstrap. All LXC containers juju creates should use 9000 as MTU for the eth0 of the container.

[Jorge Niedbalski (niedbalski)]

The fix should check if the phys interface MTU < lxc-default-mtu, then enforce the MTU on the phys interface or apply the 'post-up /sbin/ip link set eth0 mtu 9000', entry on the juju-br0 config.

[Gema Gomez (gema)]

We have tried to use the option lxc-default-mtu: 9000, on juju version 1.23.3 and did the following:

1) bootstrap to a machine
2) create an lxc containter on the bootstrap node
3) deploy ubuntu to another lxc container on the same node

The option didn't have any effect. All the interfaces ended up at MTU 1500.

[PshemK (pshem-k)]

The bug is still present in 1.25.

The fact that it only manifests itself during the subsequent machine reboots (and not during the initial build) makes it more difficult to troubleshoot - services that worked before reboot do not work after.