IPv6 MTU not applied correctly for pure IPv6 interfaces

[Victor Tapia]

[Impact]

 * In a pure IPv6 environment: if the MTU value configured for a particular interface is greater than the value of the underlying device, the MTU for such interface will be set to the value of the device.

[Test Case]

 * On a Xenial 16.04 system:
    - From this YAML: http://pastebin.ubuntu.com/22182034/
    - Curtin generates this /e/n/i: http://pastebin.ubuntu.com/22182098/
    - Use the /etc/network/interfaces and reboot
    - Check the MTU size of the device

PASS: bond2 has MTU 9202
FAIL: bond2 has MTU 1500

    ubuntu@vtapia-bond:~$ ip a | grep -E 'ens[8-9]|bond2'
    4: ens8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9202 qdisc pfifo_fast master bond2 state UP group default qlen 1000
    5: ens9: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9202 qdisc pfifo_fast master bond2 state UP group default qlen 1000
    9: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

[Original Description]

[Environment]

MAAS 2.0

[Description]

The MTU size of IPv6 must be <= device MTU (which == IPv4 MTU). This requires curtin to increase the size of the dev MTU to match the IPv6 size on pure IPv6 environments. For this scenario, the solution would be to add the following lines to e/n/i:

auto bond2
iface bond2 inet manual
    mtu 9202

[Ryan Harper (raharper)]

Are you suggesting that the existing eni which has mtu 9202 is not being applied correctly? That sounds like an ifupdown bug instead.

I'm attempting to reproduce in our vmtest to confirm.

[Ryan Harper (raharper)]

I can't recreate this issue.

I've updated our bonding test to use mtu 9202 and after booting, I run if a and check the values as you did and they're all set to 9202 (two underlying slaves and the bond itself).

[Dan Streetman (ddstreet)]

> Are you suggesting that the existing eni which has mtu 9202 is not being applied correctly? That
> sounds like an ifupdown bug instead.

yes, I opened bug 1609367 for ifupdown:
https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1609367

however, it may be difficult/impossible to fix it there, as it seems to lack the ability to conditionally do something (ip link set DEV mtu NNN) based on comparing the existing device mtu to the new ipv6-section mtu.

> I've updated our bonding test to use mtu 9202 and after booting, I run if a and check the
> values as you did and they're all set to 9202 (two underlying slaves and the bond itself).

You have to use ipv6 only to see the problem. The issue isn't with bonding at all, it's with an ipv6-only interface with a large (greater than default 1500) mtu.

[Ryan Harper (raharper)]

Thanks for the update. I changed the test to be ipv6 only and I can reproduce the problem.

We'll need to discuss the ifupdown bug as that's the *real* bug. The question will be whether it's reasonable for curtin to apply a workaround in the eni.

I'm going to run this test-case across release (precise, trusty, xenial and yakkety) to see which ones are vulnerable to this.

It appears that adding a:

post-up ifconfig <bondname> mtu <mtuvalue> to the bond stanza will successfully work-around the ifupdown bug.

[Dan Streetman (ddstreet)]

> It appears that adding a:
>
> post-up ifconfig <bondname> mtu <mtuvalue> to the bond stanza will successfully work-around
> the ifupdown bug.

unfortunately no. because the point of the ipv6 mtu is to have a lower ipv6 mtu than the device mtu, to allow ipv6 tunneling in ipv4, if that post-up is unconditionally added for all inet6 mtu params then the ipv6 mtu will never be successfully lowered to below the device mtu (whenever the device mtu is set, the ipv6 mtu is reset to the device mtu).

I go into more detail in the ifupdown bug, but two examples:

inet mtu 1500
inet6 mtu 1480
for this we want the device mtu to be 1500, and ipv6 mtu to be 1480.

# no inet mtu specified
inet6 mtu 1480
for this we also want the device mtu to stay at its default of 1500, and the ipv6 mtu to be 1480

# no inet mtu specified
inet6 mtu 9000
for this we want the device mtu to be 9000; the ipv6 will automatically be set to 9000 also.

[Ryan Harper (raharper)]

On Thu, Aug 4, 2016 at 5:28 PM, Dan Streetman <
<email address hidden>> wrote:

> > It appears that adding a:
> >
> > post-up ifconfig <bondname> mtu <mtuvalue> to the bond stanza will
> successfully work-around
> > the ifupdown bug.
>
> unfortunately no. because the point of the ipv6 mtu is to have a lower
> ipv6 mtu than the device mtu, to allow ipv6 tunneling in ipv4, if that
> post-up is unconditionally added for all inet6 mtu params then the ipv6
> mtu will never be successfully lowered to below the device mtu (whenever
> the device mtu is set, the ipv6 mtu is reset to the device mtu).
>
> I go into more detail in the ifupdown bug, but two examples:
>
> inet mtu 1500
> inet6 mtu 1480
> for this we want the device mtu to be 1500, and ipv6 mtu to be 1480.
>

OK; the current ifupdown code since 0.8.6 supports exactly this since it
changed to using sysctl for setting ipv6 mtu

>
> # no inet mtu specified
> inet6 mtu 1480
> for this we also want the device mtu to stay at its default of 1500, and
> the ipv6 mtu to be 1480
>

This should also be fine, underlying device gets default mtu of 1500, and
sysctl will handle 1480 setting for ipv6.

>
> # no inet mtu specified
> inet6 mtu 9000
> for this we want the device mtu to be 9000; the ipv6 will automatically be
> set to 9000 also.
>

The claim here is that this fails due to underlying device with default mtu
of 1500 prevents the sysctl of the device
from working. Which appears to be the case for plain interfaces:

root@x1:~# tail /etc/network/interfaces

auto ens8
iface ens8 inet6 static
    address fd0d:ffe0:5771::1:0:0/64
    mtu 9202

root@x1:~# ifup -v ens8
Parsing file /etc/network/interfaces.d/50-cloud-init.cfg
Configuring interface ens8=ens8 (inet6)
/bin/run-parts --exit-on-error --verbose /etc/network/if-pre-up.d
run-parts: executing /etc/network/if-pre-up.d/ethtool
run-parts: executing /etc/network/if-pre-up.d/ifenslave
+ [ inet6 = meta ]
+ IF_BOND_SLAVES=
+ [ ]
+ [ ]
+ [ -z ]
+ exit
run-parts: executing /etc/network/if-pre-up.d/vlan
/sbin/modprobe -q net-pf-10 > /dev/null 2>&1 || true # ignore failure.

/sbin/sysctl -q -e -w net.ipv6.conf.ens8.autoconf=0
/sbin/sysctl -q -e -w net.ipv6.conf.ens8.mtu=9202
sysctl: setting key "net.ipv6.conf.ens8.mtu": Invalid argument
/bin/ip link set dev ens8 up
/bin/ip -6 addr add fd0d:ffe0:5771::1:0:0/64 dev ens8

/lib/ifupdown/settle-dad.sh
Waiting for DAD... Done
/bin/run-parts --exit-on-error --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/000resolvconf
run-parts: executing /etc/network/if-up.d/ethtool
run-parts: executing /etc/network/if-up.d/ifenslave
+ [ inet6 = meta ]
+ [ ]
run-parts: executing /etc/network/if-up.d/ip
run-parts: executing /etc/network/if-up.d/openssh-server
run-parts: executing /etc/network/if-up.d/upstart

Now, in this scenario and post-up ifconfig <dev> mtu <mtuval> is sufficient.

I see that the post-up in the case of sharing the device name, like ens8
doesn't work as it would change ens8 mtu to the *lower* ipv6 value.

However, for vlans and bonds where there is a relationship but not direct
I think the post-up works fine:

auto ens8
iface ens8 inet static
    bond-master bo...

[Dan Streetman (ddstreet)]

> The key case to handle is not adding a post-up if the ipv6 mtu is smaller
> than ipv4 mtu (or 1500 if no mtu is set).

yes exactly.

[Ryan Harper (raharper)]

On Fri, Aug 5, 2016 at 3:33 PM, Dan Streetman <
<email address hidden>> wrote:

> > The key case to handle is not adding a post-up if the ipv6 mtu is smaller
> > than ipv4 mtu (or 1500 if no mtu is set).
>
> yes exactly.
>

Sorry it took me a bit to grasp the failure, over in the ifupdown bug I've
got a post-up hook that I think generally solves
the issue, and works on precise -> yakkety by my quick testing. Interested
in your thoughts there.

I'm inclined to mark this as invalid against curtin since there's no error
in the output, but rather a deficiency in ifupdown itself.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1609861
>
> Title:
> IPv6 MTU not applied correctly for pure IPv6 interfaces
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1609861/+subscriptions
>

[Dan Streetman (ddstreet)]

> I'm inclined to mark this as invalid against curtin since there's no error
> in the output, but rather a deficiency in ifupdown itself.

yep i agree, it can be fixed in ifupdown via bug 1609367

[Dan Streetman (ddstreet)]

To follow up, I believe it was decided in bug 1609367 to fix it in curtin for previous releases, and only fix it in ifupdown starting in yakkety or z.

[Scott Moser (smoser)]

This is fix-committed in revno 421.

[Doug Parrish (dparrish)]

ddstreet's ppa included a hotfix for this which was installed on Cust lab MAAS region controller. A machine was deployed which included a subnet defined with MTU 9202 and two NICs in a bond. After deployment, the bond showed MTU 9202 (used ifconfig), and the /e/n/interfaces file was coded in a way that a manual reboot preserved the MTU 9202 setting for the bond.

ubuntu@maas-region-controller:/etc
$ dpkg -s curtin
Package: curtin
Status: install ok installed
Priority: extra
Section: admin
Installed-Size: 11
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Version: 0.1.0~bzr424-0ubuntu1~ubuntu16.04.1
Depends: bcache-tools, btrfs-tools, dosfstools, file, gdisk, lvm2, mdadm, parted, python3-curtin (= 0.1.0~bzr424-0ubuntu1~ubuntu16.04.1), udev, xfsprogs
Description: Library and tools for the curtin installer
 This package provides the curtin installer.
 .
 Curtin is an installer that is blunt, brief, snappish, snippety and
 unceremonious.
Homepage: http://launchpad.net/curtin

[Doug Parrish (dparrish)]

curtin 0.1.0~bzr425-0ubuntu was installed out of xenial-updates and the non-default MTU does indeed get into /e/n/i.

From a deployed node's /e/n/i:
auto ens7f0
iface ens7f0 inet manual
    bond-xmit_hash_policy encap3+4
    bond-mode 802.3ad
    bond-master bond2
    mtu 9202
    bond-lacp_rate fast
    bond-miimon 100

auto ens7f1
iface ens7f1 inet manual
    bond-xmit_hash_policy encap3+4
    bond-mode 802.3ad
    bond-master bond2
    mtu 9202
    bond-lacp_rate fast
    bond-miimon 100

auto bond2
iface bond2 inet6 manual
    address fd06:ffe0:6779::2:0:9/64
    bond-xmit_hash_policy encap3+4
    bond-mode 802.3ad
    bond-slaves none
    mtu 9202
    bond-lacp_rate fast
    hwaddress ether 0c:c4:7a:b7:24:34
    bond-miimon 100

auto br-bond2
iface br-bond2 inet6 static
    address fd06:ffe0:6779::2:0:9/64
    mtu 9202
    hwaddress ether 0c:c4:7a:b7:24:34
    bridge_ports bond2