MTU not applied on private ethernet interfaces

[Andy]

== cloud-init query ==

Cloud-init isn't adding MAC addresses in the match stanza for various network interfaces in /etc/netplan/50-cloud-init.yaml, which is leading to unexpected results. Is there any reason this would happen?

== Original Description ==

Running nplan 0.30 on Ubuntu 17.10. From what I can tell, the ethernets/interface/mtu field isn't applied when testing against a private ethernet adapter.

Using a netplan yaml file (/etc/netplan/10-ens7.yaml):

---

network:
  version: 2
  renderer: networkd
  ethernets:
    ens7:
      mtu: 1450
      dhcp4: no
      addresses: [10.99.0.13/16]

---

Then running "netplan apply" (or rebooting), yields the following:

---

6: ens7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:2a:19:bc brd ff:ff:ff:ff:ff:ff
    inet 10.99.0.13/16 brd 10.99.255.255 scope global ens7
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe2a:19bc/64 scope link
       valid_lft forever preferred_lft forever

---

Is this a bug in netplan, or am I misunderstanding the YAML syntax?

Best,
Andrew

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel Axtens (daxtens)]

Still happening on 0.32~17.10.1

[Daniel Axtens (daxtens)]

So the only thing I see in the logs that looks suspicious is:

Jan 30 13:30:49 bridgevm systemd-udevd[376]: Config file /run/systemd/network/10-netplan-br0devs.link matches device based on renamed interface name 'ens8', ignoring

(my setup is explained in https://bugs.launchpad.net/ubuntu/+source/nplan/+bug/1746246)

The MTU setting is in that file, so if it is ignored it would stand to reason that the MTU would fail to apply.

It looks like this would apply to *all* renamed interfaces, which I think is lots these days.

[Andy (adavis-l)]

Confirmed that this is still an issue in bionic (beta 2, all packages updated), 0.36.1.

[Joe Gruher (joseph-r-gruher)]

Apparently this was not fixed for Ubuntu 18.04 release?

rsa@tppjoe01:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

rsa@tppjoe01:~$ uname -a
Linux tppjoe01 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

rsa@tppjoe01:~$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eno1:
            addresses: []
            dhcp4: true
            optional: true
        enp94s0f0:
            addresses:
                - 10.5.0.12/24
            mtu: 9000
            dhcp4: false
            optional: true
        enp94s0f1:
            addresses:
                - 10.6.0.12/24
            mtu: 9000
            dhcp4: false
            optional: true
    version: 2

rsa@tppjoe01:~$ sudo netplan generate

rsa@tppjoe01:~$ sudo netplan apply

rsa@tppjoe01:~$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.2.0.202 netmask 255.255.255.0 broadcast 10.2.0.255
        inet6 fe80::aa1e:84ff:fea2:9922 prefixlen 64 scopeid 0x20<link>
        ether a8:1e:84:a2:99:22 txqueuelen 1000 (Ethernet)
        RX packets 477 bytes 51423 (51.4 KB)
        RX errors 0 dropped 7 overruns 0 frame 0
        TX packets 463 bytes 56944 (56.9 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
        device memory 0x9d100000-9d17ffff

enp94s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.5.0.12 netmask 255.255.255.0 broadcast 10.5.0.255
        inet6 fe80::ee0d:9aff:fe2e:13b0 prefixlen 64 scopeid 0x20<link>
        ether ec:0d:9a:2e:13:b0 txqueuelen 1000 (Ethernet)
        RX packets 249 bytes 33951 (33.9 KB)
        RX errors 0 dropped 219 overruns 0 frame 0
        TX packets 19 bytes 1426 (1.4 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp94s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.6.0.12 netmask 255.255.255.0 broadcast 10.6.0.255
        inet6 fe80::ee0d:9aff:fe2e:13b1 prefixlen 64 scopeid 0x20<link>
        ether ec:0d:9a:2e:13:b1 txqueuelen 1000 (Ethernet)
        RX packets 249 bytes 34056 (34.0 KB)
        RX errors 0 dropped 218 overruns 0 frame 0
        TX packets 19 bytes 1426 (1.4 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 160 bytes 11012 (11.0 K...

[Daniel Axtens (daxtens)]

It should work if you match by MAC address. It's a messy interaction with
udev, as I understand it. Could you try matching by MAC and seeing if that
works?

We should definitely document it better.

[Joe Gruher (joseph-r-gruher)]

Yup, adding a match on MAC seems to work. It would be nice if whatever sets up the default /etc/netplan/50-cloud-init.yaml included the match on MAC?

rsa@tppjoe03:~$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.2.0.203 netmask 255.255.255.0 broadcast 10.2.0.255
        inet6 fe80::aa1e:84ff:fea2:989c prefixlen 64 scopeid 0x20<link>
        ether a8:1e:84:a2:98:9c txqueuelen 1000 (Ethernet)
        RX packets 335 bytes 32455 (32.4 KB)
        RX errors 0 dropped 6 overruns 0 frame 0
        TX packets 405 bytes 60596 (60.5 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
        device memory 0x9d100000-9d17ffff

enp94s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.5.0.13 netmask 255.255.255.0 broadcast 10.5.0.255
        inet6 fe80::ee0d:9aff:fe2e:13d0 prefixlen 64 scopeid 0x20<link>
        ether ec:0d:9a:2e:13:d0 txqueuelen 1000 (Ethernet)
        RX packets 226 bytes 29676 (29.6 KB)
        RX errors 0 dropped 204 overruns 0 frame 0
        TX packets 12 bytes 936 (936.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp94s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.6.0.13 netmask 255.255.255.0 broadcast 10.6.0.255
        inet6 fe80::ee0d:9aff:fe2e:13d1 prefixlen 64 scopeid 0x20<link>
        ether ec:0d:9a:2e:13:d1 txqueuelen 1000 (Ethernet)
        RX packets 227 bytes 29900 (29.9 KB)
        RX errors 0 dropped 204 overruns 0 frame 0
        TX packets 12 bytes 936 (936.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 144 bytes 9808 (9.8 KB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 144 bytes 9808 (9.8 KB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

rsa@tppjoe03:~$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eno1:
            addresses: []
            dhcp4: true
            optional: true
        enp94s0f0:
            match:
                macaddress: ec:0d:9a:2e:13:d0
            addresses:
                - 10.5.0.13/24
            mtu: 9000
            dhcp4: false
            optional: true
        enp94s0f1:
            match:
                macaddress: ec:0d:9a:2e:13:d1
            addresses:
                - 10.6.0.13/24
            mtu: 9000
            dhcp4: false
            optional: true
    version: 2
rsa@tppjoe03:~$ sudo netplan generate
rsa@tppjoe03:~$ sudo netplan apply
rsa@tppjoe03:~$ ifconfig
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 10.2.0.203 netmask 255.255.255.0 broadcast 10.2.0.255
        inet6 fe80...

[Daniel Axtens (daxtens)]

Glad that helped. I'm confused as to why cloud-init doesn't add the mac
addresses - it does in my VMs. Let's ask them; I'll add them to the bug.

[Ryan Harper (raharper)]

Can you run cloud-init collect-logs in your instance and attach the tarfile (cloud-init.tar.gz in the current working directory).

[Joe Gruher (joseph-r-gruher)]

Adding requested file.

[Andy (adavis-l)]

Matching by MAC address corrects the MTU issue on both Ubuntu 17.10 and Ubuntu 18.04.

I noticed one side effect though - this happened on fresh Ubuntu 17.10 and Ubuntu 18.04 installs. After adding the "match: ... macaddress: ..." parts to the YAML file, running "netplan apply" caused the "ens7" device to be renamed to "eth0". I'm not sure if this is related to my VM configuration, or if it's a bug in netplan. After rebooting, the "ens7" device name was used and the MTU was set properly. Details follow.

----

My "/etc/netplan/10-ens7.yaml" contains:

 network:
   version: 2
   renderer: networkd
   ethernets:
  ens7:
    match:
   macaddress: 5a:00:01:7d:18:9a
    mtu: 1450
    dhcp4: no
    addresses: [10.1.96.4/20]

Before running "netplan apply":

 3: ens7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
  link/ether 5a:00:01:7d:18:9a brd ff:ff:ff:ff:ff:ff

 ...

After running "netplan apply":

 4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
  link/ether 5a:00:01:7d:18:9a brd ff:ff:ff:ff:ff:ff
  inet 10.1.96.4/20 brd 10.1.111.255 scope global eth0
     valid_lft forever preferred_lft forever
  inet6 fe80::5800:1ff:fe7d:189a/64 scope link
     valid_lft forever preferred_lft forever

Dmesg shows:

 [ 585.985711] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
 [ 586.812306] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Journalctl shows:

 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: ens3: Gained IPv6LL
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: Enumeration completed
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: ens3: Link is not managed by us
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: lo: Link is not managed by us
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: eth0: IPv6 successfully enabled
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: lo: Link is not managed by us
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: ens3: DHCPv4 address 45.77.216.215/23 via 45.77.216.1 unit='dbus-org.freedesktop.hostname1.service' requested by ':1.9' (uid=100 pid=925 comm="/lib/systemd/systemd-networkd " label="unconfined")
 May 09 23:03:44 ubuntu1804 systemd-networkd[925]: ens3: Configured
 May 09 23:03:45 ubuntu1804 systemd-networkd[925]: eth0: Gained carrier
 May 09 23:03:47 ubuntu1804 systemd-networkd[925]: eth0: Gained IPv6LL
 May 09 23:03:47 ubuntu1804 systemd-networkd[925]: eth0: Configured

[Daniel Axtens (daxtens)]

Hi Andy,

This is a known issue - after giving backends a chance to bring up network
interfaces, netplan apply will iterate over interfaces that are still down,
unbind them from their driver and rebind them. This can cause renaming of
the interface.

In general, adding "set-name: ens7" to your YAML file should prevent it
happening in the future. Note that netplan apply won't re-rename it if the
interface is UP; netplan won't unplug the device unless you bring it down
first. You can work around this by rebooting.

(For this general issue I opened
https://bugs.launchpad.net/netplan/+bug/1753868 - although it was one of
the first netplan bugs I opened so it's a bit light on technical detail
that I'd add if I were opening it now.)

Regards,
Daniel

[Andy (adavis-l)]

Hi Daniel,

Thank you for clarifying! I didn't see that was already a known issue.

Best,
Andrew

[EOLE team (eole-team)]

I have the same here.

Netplan creates .link with

[Match]
OriginalName=ens4

Which can't be used since the network interface was already renamed

[...]
Jul 02 15:46:09 eolebase kernel: virtio_net virtio1 ens4: renamed from eth0
[...]
Jul 02 15:46:09 eolebase systemd-udevd[310]: Config file /run/systemd/network/10-netplan-ens4.link matches device based on renamed interface name 'ens4', ignoring

I see two options:

1) Match on the MACAddress
2) Define the link properties in the .network file

Regards.

[zach (zpincus)]

I just got bit by this one too.

This is frustrating since the top two (for me) Google hits for "ubuntu netplan mtu" both come from official Ubuntu sources, and both provide broken solutions:
https://blog.ubuntu.com/2017/12/01/ubuntu-bionic-netplan
https://blog.ubuntu.com/2017/07/05/quick-and-easy-network-configuration-with-netplan

The first one is especially bad because it seems to suggest that, on Bionic, one can set an MTU by only matching on one of the "Predictable Network Interface Names". Which of course has never worked on Bionic, per this bug.

Only after clicking through various forum links does one come to this bug report. Nowhere does any of this appear in the ubuntu or netplan documentation. I mean, if it's been known for a year or more that many aspects of netplan don't work as-advertised on renamed devices, and by default almost all network devices get renamed these days, this seems like a major problem for the default network configuration tool, no?

Last, needing to match on MAC addresses to work around issues caused by "predictable interface names" is particularly perverse! Recall that one of the main rationales for such interface renaming (instead of eth0 etc) was that MAC addresses aren't necessarily static on embedded / virtualized systems:
https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

[Joshua Powers (powersj)]

Logs were added for cloud-init, so marking the cloud-init task back to 'new'.

[jd (jeff-dyke)]

I'm having a similar issue where setting the MTU to 1500 and using a match on mac address does not stick. This is on a EC2 T2.Medium instance. Actual configuration from a dev server:
network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: 12:0f:ae:49:5d:06
      mtu: 1500
      dhcp4: true
      nameservers:
        search: [ devbuilds.vpc, ec2.internal ]

$> cat /var/run/systemd/network/10-netplan-eth0.link
[Match]
MACAddress=12:0f:ae:49:5d:06

[Link]
WakeOnLan=off
MTUBytes=1500

$> cat /var/run/systemd/network/10-netplan-eth0.network
[Match]
MACAddress=12:0f:ae:49:5d:06

[Network]
DHCP=ipv4
Domains=devbuilds.vpc ec2.internal

[DHCP]
UseMTU=true
RouteMetric=100

$> ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 12:0f:ae:49:5d:06 brd ff:ff:ff:ff:ff:ff
# manually set MTU
$> sudo ip link set dev eth0 mtu 1500
$> ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 12:0f:ae:49:5d:06 brd ff:ff:ff:ff:ff:ff
$> sudo netplan generate
$> sudo netplan apply
$> ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 12:0f:ae:49:5d:06 brd ff:ff:ff:ff:ff:ff

The last netplan commands could be replaced with a reboot with the same result. This configuration seemed to help others, so hopefully i'm simply missing something or perhaps this is related to EC2?

Attaching output of cloud-init

[Mathieu Trudel-Lapierre (cyphermox)]

That is a different issue we're solving elsewhere. You should be able to override the UseMTU= field.

[Mathieu Trudel-Lapierre (cyphermox)]

Clearing up bug statuses: We'll fix this for 18.04 and above, but first it needs to land in the development release. Won't Fix is for the releases before 18.04.

[Mathieu Trudel-Lapierre (cyphermox)]

Please see if this is solved by the fix for https://bugs.launchpad.net/netplan/+bug/1807273 (setting use-mtu: false)

[Chad Smith (chad.smith)]

Marking this invalid for cloud-init per netplan symptoms 'fixed' in https://bugs.launchpad.net/netplan/+bug/1807273 look like it might be the same case here with no change to cloud-init

[Lars Erik Pedersen (pedersen-larserik)]

Can we have a status on this? I can also confirm that the suggested workaorund works, but this should be fixed anyhow, right?

[Jean-Daniel Dupas (xooloo)]

Still broken on 18.04.03 (netplan 0.97).

[Michele Giacomoli (michele-giacomoli)]

Still broken on 18.04.3 with netplan 0.98-0ubuntu1~18.04.1 and systemd 237-3ubuntu10.31.

With dhcp set to false, use-mtu seems to be useless. The only affordable workaround I found is to create an override .link file that only matches by mac address and set the MTU. For example:

[Match]
MACAddress=aa:bb:cc:dd:ee:ff

[Link]
MTUBytes=9000

saved as /etc/systemd/network/10-cmonsystemdsetmtuforme.link. Reboot (maybe also "systemctl restart systemd-networkd" works) and mtu should be set.

Hoping for a fix in the near future.

[Stefan Greiner (sgreiner)]

Any updates for bionic? We would strongly need this fixed.

[Patrick Domack (patrickdk)]

This bug is also affecting 20.04

[Kraut.Hosting (kraut.hosting)]

@cyphermox Need to confirm this on focal daily ... we still need a "match: macaddress" for setting MTU.
This is really a show stopper when cloning servers with virtualization and containerization an new MAC.
Working with VLANs this is an essential setting for getting networking before config mgmt can template.

[Jean-Daniel Dupas (xooloo)]

Still broken in 20.04 beta. This is a real shame…

[Jean-Daniel Dupas (xooloo)]

What prevent the generator to use Match/Name instead of Match/OriginalName by default in .link file like it does for generated .network files ?

As the few properties that may require a match on the original name (like set-name) should already use a match directive anyway, so it should not be an issue.

[Steve Langasek (vorlon)]

Why does this bug title refer to "private" ethernet adapters? It sounds like you're saying the problem is much more general than this.

[Lukas Märdian (slyon)]

Doing some experiments this seems to be working with netplan.io 0.100 in Groovy. No matter if 'match.macaddress' is set or not on my local eth interface.

As far as I can tell the fix was released with netplan 0.99 in Focal final and was SRU'ed to Bionic as well. (https://github.com/CanonicalLtd/netplan/commit/77fcd4dbf77092608021baf5be5e4ae237f7de86)

Marking 'Fix released'. Please re-open if it does not work for you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3034