Interface MTU management across MAAS/juju

[James Page]

Context:

juju + MAAS deployed OpenStack environment, misc services deployed under LXC on the bootstrap node, interfaces configured for jumbo frames - note that I had to manually set the LXC container interfaces to mtu 9000 before the bridge would do the same.

Action:

Reboot one of the containers; MTU on br0 resets from 9000 -> 1500.

This feels like more of a 'we need a better way to orchestrate MTU configuration across different services' so raising tasks for MAAS and Juju as well.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: lxc 1.0.5-0ubuntu0.1
ProcVersionSignature: User Name 3.13.0-24.47-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.3
Architecture: amd64
Date: Tue Aug 12 13:26:00 2014
KernLog:

SourcePackage: lxc
UpgradeStatus: No upgrade log present (probably fresh install)
defaults.conf:
 lxc.network.type = veth
 lxc.network.link = lxcbr0
 lxc.network.flags = up
 lxc.network.hwaddr = 00:16:3e:xx:xx:xx
lxcsyslog:

[Serge Hallyn (serge-hallyn)]

Hi James,

does setting lxc.network.mtu = 9000 in the container configuration file work?

Depending on how juju creates the containers, you may be able to simply add that line to the bottom of /etc/lxc/default.conf to get the desired result.

[Kapil Thangavelu (hazmat)]

i've seen reference to you can't really set a bridge mtu, you have to set it for all the containers on the bridge, ie. bridges take the lowest mtu of their interfaces. so instead we have to set all the containers mtu.

[James Page (james-page)]

why is this bug incomplete? as far as I can see there is sufficient information to triage this...

Yes - you have to set the mtu on the lxc container interfaces to match the bridge, otherwise it drops to the lowest MTU of any attached veth.

For now I've patched the config files to set the mtu correctly - but as soon as you juju add-unit --to lxc:XX it all collapses again.

For OpenStack environments running overlay networks, setting interface MTU's on hosts is absolutely critical.

[Serge Hallyn (serge-hallyn)]

@James,

the bug was incomplete because the question in comment #2 was never answered until comment #4.

So it sounds like something juju needs to do through lxc config - if there is anything sane that you can think of htat lxc can do to help, please let us know.

[James Page (james-page)]

I've re-titled this bug to be about effective interface MTU management across MAAS and Juju; enabling Jumbo frames is something that we should support in our core tooling and is critical for the successful operation of a Neutron based OpenStack Cloud.

Combine this with LXC container spread across physical servers and right now this is fiddly and error prone.

[Dimiter Naydenov (dimitern)]

Proposed a fix with http://reviews.vapour.ws/r/686/ - before starting a container, the host's primary physical NIC's MTU value is detected and applied to the container's veth device.

[Ian Booth (wallyworld)]

Marked as Fixed Released, does it still occur?

[Mark Shuttleworth (sabdfl)]

Do we need to take into account any overhead being added by the protocol stack along the way?

For example, if the underlying MAAS machine can be told to use 9000 byte MTUs and jumbo frames, then each layer up could adopt that, but might need to chop off some of that for protocol / tunnel / encapsulation overhead. I've seen cases where your "internal" interface needs to have, say 20 byte smaller MTUs than the host. Perhaps THAT's the bit that we should focus on specifying. So MAAS gets told "just jumbo frames here" and then everything else focuses on understanding and allowing for overhead, but maximising MTU subject to that overhead.

[Dimiter Naydenov (dimitern)]

See also the related bug 1442257 which has a proposed fix.