MTU size defined on /etc/netplan/50-cloud-init.yaml not applied

Bug #1804861 reported by Alvaro Uría on 2018-11-23
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
netplan
Undecided
Unassigned
maas (Ubuntu)
Undecided
Unassigned
netplan.io (Ubuntu)
Undecided
Unassigned

Bug Description

netplan version: 0.36.3
Ubuntu Bionic

"""
network:
    version: 2
    vlans:
        bond0.810:
            addresses:
            - 10.20.33.3/29
            id: 810
            link: bond0
            mtu: 1500
            nameservers: *id001
"""

Note: bond0 and its slaves have mtu=9000 configured (there are other vlans using 9000, too).

On netplan source code deployed by the package, only "migrate.py" has references to "MTU", however, a command does not exist for "migrate".

I also tried to add /etc/udev/rules.d/70-net.rules:
"""
SUBSYSTEM=="net", ACTION=="add", KERNEL=="bond0.810", ATTR{mtu}=="9000", ATTR{mtu}="1500"
"""

But it doesn't get applied.

Calvin Hartwell (calvinh) wrote :

subscribed ~field-critical as there is no apparent work around and this is blocking a deployment in final phase to bootstack handover. Currently broken in 18.04 LTS release.

Seems to work here. With the config below; 'ip link' does show the bond device and the physical interfaces having a MTU of 9000, and the two VLANs having a MTU of 1500, as per the configuration.

What could be the difference that breaks this deployment?

network:
    version: 2
    ethernets:
        ens6:
            set-name: ens6
            match:
                macaddress: 52:54:00:25:ba:4f
            mtu: 9000
        ens7:
            set-name: ens7
            match:
                macaddress: 52:54:00:02:86:3b
            mtu: 9000
    bonds:
        bond0:
            interfaces:
            - ens6
            - ens7
            macaddress: 00:de:ad:be:ef:01
            mtu: 9000
            parameters:
                down-delay: 0
                lacp-rate: fast
                mii-monitor-interval: 100
                mode: 802.3ad
                up-delay: 0
                transmit-hash-policy: layer3+4
    vlans:
        bond0.804:
            mtu: 1500
            id: 804
            link: bond0
            addresses:
                - 10.20.16.50/24
        bond0.1:
            mtu: 1500
            id: 1
            link: bond0
            addresses:
            - 10.3.67.31/20
            gateway4: 10.3.67.1
            nameservers:
                addresses:
                - 10.3.67.22
                search:
                - cloud

Even adding more VLANs in this setup with alternating MTU values, the values do appear to be applied as per the config.

What version of systemd is in use on this system?

Looks like I couldn't reproduce because I have been setting a custom MAC on the bond interface rather than using the MAC from one of the underlying devices.

If an existing physical MAC is used for the bond, then the matching in systemd/udev works such that MTU will be applied to all the interfaces with that MAC. If not all of them get the right MTU, it's because this may end up being racy.

There are two solutions to this:

 - match on macaddress *and* driver (or name, but driver should be more solid) on the physical interfaces, such that the MTU is only applied to the devices that match both the MAC address and driver tuple exactly.
 - set a custom MAC that isn't the MAC address from one of the underlying interfaces on the bond

I don't think we can address this in netplan programmatically, since that would break the matching story if we were to implicitly add matching rules without them appearing in the configuration file.

Peter Sabaini (peter-sabaini) wrote :

Mathieu,

systemd version is straight from Bionic:

ii systemd 237-3ubuntu10.9 amd64 system and service manager

Peter Sabaini (peter-sabaini) wrote :

Mathieu,

for clarification, does this mean the bug is in systemd then, and you're suggesting for option 1 (matching on macaddr) to implement this in systemd?

What I'm suggesting right now is only a workaround:

When deploying the system in MAAS, set the MAC of the bond to something rather than using the default. You may pick a MAC address within the 52:54:*:*:*:* address space, or probably b2:31:91:*:*:* (which is what gets used if the MAC is completely unset in the netplan config).

I think there are various pieces competing here:
 - MAAS is trying to be clever, suggests a default value which is the MAC of one of the physical interfaces as a default value.
 - systemd allows matching on things other than MAC, driver, or name, but those are pretty much useless for our purposes (Type= is unset on physical ethernet devices, Path= is unset on VLANs/bonds)
 - netplan could second-guess the user and attempt at matching more strictly by device type (but see above, impossible due to a limitation in systemd/udev); at least should warn that the matching isn't unique.

I've been compiling a better example to show the behavior.

I compiled three different cases and their behavior (see attached). In each case, I have included the netplan config, contents of files in /run/systemd/network and the output of 'ip link':

- Case A: MAC is unset in netplan config (I do not think MAAS allows this right now):
 - here, the system behaves correctly. MTUs are correctly set, devices gain a MAC address in b2:31:91:*. I am not sure yet what assigns it that, I think it's the kernel.

- Case B: using a custom MAC address for the new bond interface (possible in MAAS)
 - here, the system again behaves correctly. MTUs are correctly set, and devices all get the same MAC address as defined. I used "52:54:45:be:de:ad"; I believe any MAC under the range for virtualized devices should work (52:54:*), obviously any valid MAC that doesn't already exist on the network is fine to use.

- Case C: using the MAC of the first physical interface (default in MAAS)
 - The VLAN under the bond gets the wrong MTU (9000), due to the imprecise matching. All devices get the same MAC, the one originally set to ens6 in my example (52:54:00:25:ba:4f).

Ryan Harper (raharper) wrote :

Can you attach the complete netplan config? If this is maas deployed then we'll have a /etc/netplan/50-cloud-init.yaml.

And if you can also please on the node:

1)
mkdir /etc/systemd/system/systemd-networkd.service.d/
echo -e "[Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug" > /etc/systemd/system/systemd-networkd.service.d/10-debug.conf

2) systemctl daemon-reload

3) reboot

And then after rebooting, collect journalctl -b 0 and /run/systemd/network/*

Peter Sabaini (peter-sabaini) wrote :

Ryan,

I've put diagnostics up here: https://private-fileshare.canonical.com/~sabaini/lp1804861/lp1804861-netplan.tar.gz

Note this node is Maas+juju deployed, I believe juju puts a netplan of its own there. Also added output of ip link list

Thanks,
peter.

tags: added: id-5bfdb43fc0744c8fae37a20e

Thanks for the information. I can confirm what Mathieu is seeing in
Case C. Initially when I deploy your netplan config, all of the MTU
settings are correct
due to the .link files already being triggered; it's somewhat of a
race with udev and when the underlying interfaces come up.
The juju scenario though almost ensures that we'll hit a case where
the vlan has the correct MTU initially, juju eventually calls netplan
apply
which can trigger the system to apply .link files which for the bonds,
will match additional interfaces due to use of the underlying MAC
address.

As suggested, one possible workaround is for MAAS to generate MAC
values for bridges/bonds and ensure they're not one of the physical
device
MAC values; this would ensure .link rules which use MAC matching won't
apply. I don't know the ramifications of that w.r.t deployments of
switches
and other things which sometimes want/need to know what MAC will show
up on which port etc.

On the netplan side, if a bond or bridge is assigned a MAC from one of
the ethernet devices, it could emit additional properties in the
ethernet
.link file to prevent the .link from applying to devices other than
the actual bond members.

In the two bond member's .link files:

# cat 10-netplan-enp4s0f0.link
[Match]
MACAddress=00:16:3e:4e:89:25
Type=!vlan bridge bond

[Link]
WakeOnLan=off
MTUBytes=9000
On Wed, Nov 28, 2018 at 7:50 AM Francis Ginther
<email address hidden> wrote:
>
> ** Tags added: id-5bfdb43fc0744c8fae37a20e
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1804861
>
> Title:
> MTU size defined on /etc/netplan/50-cloud-init.yaml not applied
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1804861/+subscriptions

I disagree.

We'd be working around a real design issue in systemd, rather that possibly matching against Type=ethernet (if that existed). I think we should instead prioritize on avoiding this broken case in the other parts of the networking stack, and make sure we fix systemd to provide the right matching Type.

Nothing guarantees that vlan, bridge, and bonds are the only types of devices affected and we're second-guessing the admin otherwise: I rather we match precisely on what was added as matches in the netplan YAML, in the case where you set the MAC to the same on all of a stack because you really *did* want to apply some of the same options to all of these devices.

Ryan Harper (raharper) wrote :

On Tue, Dec 4, 2018 at 10:30 AM Mathieu Trudel-Lapierre <
<email address hidden>> wrote:

> I disagree.
>
> We'd be working around a real design issue in systemd, rather that
>

While, I agree that having systemd-udev to apply MTU to devices is
less than ideal; I don't see having systemd or networkd growing
support for applying MTU values intelligently in the face of how several
key network devices in linux work (bonds and bridges utilize member
MAC values to function) happening any time soon.

This leaves netplan systems wanting to use bonds/bridges as they have
in Xenial/ifupdown regressed in functionality and requires (at least in the
MAAS case) existing products to update to work around netplan/networkd
MAC matching behavior that differs from what worked on Xenial.

Even if we "fixed" MAAS, there may be additional systems which generate
netplan configurations and expect that a bond or bridge will use the MAC
from one of the members. This impacts how switches and other off-system
devices are configured (port filtering by mac).

> possibly matching against Type=ethernet (if that existed). I think we
> should instead prioritize on avoiding this broken case in the other
> parts of the networking stack, and make sure we fix systemd to provide
> the right matching Type.
>

> Nothing guarantees that vlan, bridge, and bonds are the only types of
> devices affected and we're second-guessing the admin otherwise: I rather
>

I don't think we're second guessing anything; what I'm suggesting is that
when we examine the netplan yaml we have references to the bond or bridge
members, and we can determine what TYPE they are (or some other matching
element) and insert those into the bond match section to ensure it only
matches
for the interfaces with which the bond is configured.

The problem is that DEVTYPE is unreliable at best. Using the MAC of one of the member interfaces is fine, but only if you really know what you're doing.

I think we're otherwise back to matching by name instead of by MAC address (or matching both), if you want to match "like ifupdown was doing".

This is why we should fix networkd -- I don't think it's a difficult thing to do at all to say, have it grow an actual value for DEVTYPE rather than null, nor would be be complicated to then write Type=ethernet in the .link files generated, provided that the syntax also read 'match: { type: ethernet }'... or perhaps even not. We can do this fix ourselves, I'm just pointing out that it has the potential of being a bit disruptive, because ethernets have long had no DEVTYPE value in udev (it's been a PITA for a while).

*Then* we can improve the matching logic in netplan, but I don't think we should do that before we have the right facilities in systemd to do so.

In the interim; I still do think it would be best for maas/juju and whatever not to assume that using the MAC address for the first phy is the right thing to do.

And before then, workaround is to explicitly define a MAC address for the device when defining the bond or bridge rather than using defaults.

The logic is simple: the kernel and systemd already do the right thing if no MAC is set at all, so it could just as well be omitted completely from the netplan syntax generated by the different install tools. If you must write one, then you can always come up with one in a range that will not clash with other devices on the network.

I'm not against finding a suitable workaround in netplan, but so far I haven't seen anything that seemed sufficiently solid, and let's be honest: those remain workaround to paper over other issues that we really should be fixing instead.

Ryan Harper (raharper) wrote :
Download full text (4.5 KiB)

On Tue, Dec 4, 2018 at 12:31 PM Mathieu Trudel-Lapierre <
<email address hidden>> wrote:

> The problem is that DEVTYPE is unreliable at best. Using the MAC of one
> of the member interfaces is fine, but only if you really know what
> you're doing.
>
> I think we're otherwise back to matching by name instead of by MAC
> address (or matching both), if you want to match "like ifupdown was
> doing".
>
> This is why we should fix networkd -- I don't think it's a difficult
> thing to do at all to say, have it grow an actual value for DEVTYPE
>

DEVTYPE is a kernel device property that udev exports into systemd/networkd

so it's not an easy fix to networkd; it's kernel patches for all net
devices.

> rather than null, nor would be be complicated to then write
> Type=ethernet in the .link files generated, provided that the syntax
> also read 'match: { type: ethernet }'... or perhaps even not. We can do
> this fix ourselves, I'm just pointing out that it has the potential of
> being a bit disruptive, because ethernets have long had no DEVTYPE value
> in udev (it's been a PITA for a while).
>

Hrm, you seen to agree that we don't have this property from udev and it's
going to be
a problem, so I'm confused about what you're proposing to do in networkd.

>
> *Then* we can improve the matching logic in netplan, but I don't think
> we should do that before we have the right facilities in systemd to do
> so.
>

I think for any of the devices that may be members of a composed device, we
can find out udev properties which can be used in the Match section for
.link
files to filter out any device besides the composed device members.

> In the interim; I still do think it would be best for maas/juju and
> whatever not to assume that using the MAC address for the first phy is
> the right thing to do.
>

It's not that maas/juju assume it's the right thing to do, it's what
happens in linux
networking by default (in the absence of providing a dedicated MAC value
for the
bond or bridge). This is the convention that currently doesn't work under
netplan
when it comes to applying MTU due to the [Match] were generating.

>
> And before then, workaround is to explicitly define a MAC address for
> the device when defining the bond or bridge rather than using defaults.
>

I'll let the field folks comment on whether that's something that's doable
I suspect in a case-by-case basis it may, but generally this adds burden
to deployments which work fine on Xenial, but not on Bionic.

> The logic is simple: the kernel and systemd already do the right thing
> if no MAC is set at all, so it could just as well be omitted completely
> from the netplan syntax generated by the different install tools. If you
>

Even if this works, it's not an easy thing to ensure that the network
configuration
that MAAS sends to deployed systems omits a mac address for the bond.
At minimum it's a big SRU on all of the supported releases.

> must write one, then you can always come up with one in a range that
> will not clash with other devices on the network.
>
> I'm not against finding a suitable workaround in netplan, but so far I
> haven't seen anything that seemed sufficiently solid, and let'...

Read more...

It doesn't matter which process we use, it will be a big SRU regardless.

Also, of course it's an easy fix in MAAS, and it obviously works. I've shown that setting no "macaddress" field in the netplan syntax does what it should. Just don't specify a default for that value, and don't write the macaddress field in the generated config.

The behavior that happens in kernel is largely irrelevant; because that would be skipped anyway through the use of networkd, which will do its own MAC address handling. You do not need to, and definitely should not, attempt to be clever and provide default values for optional fields unless you really know what the behavior will be, or somehow really need to override the behavior of what is authoritative on the network management -- that's networkd and netplan.

Sure, the current kernel behavior is to use the first phy, that's fine, but not something that needs to be emulated in userland as well. It could have been fine, if it wasn't for networkd matching devices incorrectly, which is (the matching) a bug that needs to be fixed. Similarly, the fact that the kernel needs patching shouldn't be a blocker for fixing the issue (missing DEVTYPE).

Now, I *am* looking at a suitable "workaround" in netplan, but as expressed before, this is a *workaround* to work around MAAS/juju/curtin writing default values it doesn't need to write, based on assumptions that may be correct for the kernel, but do not need to be repeated in userland. I don't have the solution right now, but the options for matching are basically limited to MACAddress=, OriginalName=, Path=, Driver= and Type=. Others aren't useful.

Working from this, MACAddress= is already out of the question -- it's where the issue comes from.

OriginalName= is largely irrelevant, we don't necessarily know this to begin with.

Type= is not useful at all; DEVTYPE= isn't set for ethernets, as discussed earlier.

Path= is not especially great, no useful ID_PATH available on the bond, but physical devices do set something (ID_PATH=pci-0000:00:06.0 / ID_PATH=pci-0000:00:07.0). Possibly could be used, but that will break/become extremely complicated if we try to match against a glob.

To be clear: my concern isn't just this particular use case of deployment via MAAS. I want this to work correctly in all cases, and I think the least complicated way to get to that point is to not specify the MAC address when defining the bond, or defining one that is different from the permanent physical MAC of the underlying devices, or explicitly matching on other values than the MAC address.

The point is: if I just workaround in netplan, the other issues won't get fixed. Please make sure you prioritize fixing the various configuration generators such that they stop making the wrong assumptions they are making.

Mike Pontillo (mpontillo) wrote :

This is a long thread; sorry for adding fuel to the fire. But I have a some questions/comments.

(1) I feel like if we don't fix how matching works for 'ethernets' in Netplan, we're not mapping the user's intent to an appropriate configuration. The user is saying "I have two physical interfaces. I want the one that has MAC X to have name A. I want the one that has MAC Y to have name B." They aren't modeling anything about what they want the bond configuration to be. They're telling Netplan how to identify a particular interface.

That is, users won't be thinking in terms of the resulting match rules. They're thinking in terms of how we've presented the model in Netplan.) If Netplan determines that the configuration would be ambiguous, it's free to write additional match rules that the user didn't specify. (Though I do agree that could be painful to implement.)

I would love it if MAAS didn't need to send in the MAC at all. Unfortunately, I don't think this is viable long-term, as the kernel, plus algorithms for determining interface names, will change over time. The only stable match criteria we have been able to rely on across all distros and releases that MAAS supports is the MAC.

(2) We cannot easily add driver matching to MAAS. This would be tricky/undesired for a number of reasons:
 - The driver the kernel uses could change over time.
 - This would be a schema change that we could land for MAAS 2.6 at the earliest. (Unless we employ inefficient hacks to avoid that.)
 - This would require adding at least a post-commissioning hook to MAAS, and possibly another commissioning script, in order to gather the additional match parameters.

That said, I think to some degree it's not great for MAAS to only match on MAC address. (This can get tricky, for example, if we're deploying to a bridge NIC where each port has the name MAC but appears as multiple interfaces.)

(3) The reason MAAS assigns a physical MAC to a bond interface is due to issues that have occurred in the past that require it. For example, you may have a DHCP server that begins handing out a different lease to a bond, when we expected it to get the same address as the physical interface (for example, if the machine was recently PXE booted from one of the bond interfaces). For this (and possibly other reasons that are escaping me) our users generally aren't fans of unstable MAC addresses across deployments of the same hardware.

My $0.02: if Netplan can gather any additional criteria it needs to apply more specific rules at "apply" time to "ethernets" defined based solely on the MAC, that would be preferred to the current behavior.

Steve Langasek (vorlon) wrote :

Hi Mike,

I broadly agree with your comments, but I want to note that:

On Wed, Dec 05, 2018 at 02:05:37AM -0000, Mike Pontillo wrote:
> I would love it if MAAS didn't need to send in the MAC at all.
> Unfortunately, I don't think this is viable long-term, as the kernel,
> plus algorithms for determining interface names, will change over time.
> The only stable match criteria we have been able to rely on across all
> distros and releases that MAAS supports is the MAC.

We are still in the midst of a multi-meta-cycle transition to migrate toward
the use of stable kernel device naming, but once that's completed there is
no reason to believe kernel device names will change further. Furthermore,
on any image where we have netplan available, the default policy for the
kernel is to use predictable network device names. So I don't think this
should be an argument for forcing MACs going forward for anywhere that
you're using netplan.

Mike Pontillo (mpontillo) wrote :

Thanks Steve; that's good to know. I think that will be very useful, especially when deploying a machine that has hardware such as switch ports, which may show up as separate interfaces but share a MAC. I'm open to adding driver information in future releases.

We still need to be careful on this, because users may be deploying custom images without stable kernel device names, or may be deploying entirely non-Linux OSes. I expect that such deployments would simply ignore Netplan keys they don't understand, such as kernel device names. So I expect that MAAS could one day include the device name /and/ MAC as match criteria.

MAAS picks up information at commissioning time about the system. This would include the NIC driver, which could just as well be used as an additional matching criterion in netplan YAML.

There's a limit to how much netplan can and should try to figure out what one means: we're supposed to do *translation* of configuration -- sure, we could check the running system at the time to try to make the config more precise, but that only helps *iff* the user wasn't actually precisely requesting what they wanted. Anything we add to what's written in the actual netplan YAML needs to be carefully weighed against whether we risk actively going against what the user has written. The user knows best what their network is like.

This is why I'm strongly suggesting that in addition to what I'm adding in netplan, changes are made to MAAS to ensure the config is as precise as it possibly could be, while also not including default values on optional fields without the admin making a conscious decision on adding these values (ie. default MAC for a bond). MAAS knows what the MAC for a physical device is, what its name should be, and even down to the driver. Sure, any of those could change, but that's not different from any other environment: devices fail, hardware needs replacement. The MAC would have changed anyway. At least with MAAS you can re-commission and re-deploy systems easily when that happens.

I've added https://github.com/CanonicalLtd/netplan/commit/a27122bc8d8e066b1a90a7fd8d65342e8b906a8e to netplan work mitigate the current issue. This shouldn't be considered an end-all solution -- I still think other pieces of the network stack need changing.

It's as much as I am comfortable adding to netplan to mitigate the problem. There isn't enough information coming from the YAML to figure out more details about the interface, nor a non-hackish way to retrieve the information from the system when networkd will do the matching later.

Changed in netplan:
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netplan.io - 0.95

---------------
netplan.io (0.95) disco; urgency=medium

  * New upstream release:
    - Added support for WPA Enterprise / 802.1x authentication (LP: #1739578)
    - Added support for setting up IP tunnels; supporting the types: ipip,
      gretap, VTI, ISATAP (NetworkManager only), sit, gre, ipip6 and ip6ip6.
      + Fixes sit (ipv6) tunnels using Hurricane Electric (LP: #1799487)
    - Add support to override networkd UseMTU setting (LP: #1807273)
    - Generate output files in dependency order
    - Refactored unit and integration tests, along with various cleanups.
    - Add DHCP overrides to control route usage and default metric for DHCP
      routes. (LP: #1776228)
    - Mitigate against bad matching on devices behind bonds then they share
      the same MAC from a physical interface. (LP: #1804861)
    - Added snapcraft.yaml.

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 14 Dec 2018 16:04:05 -0500

Changed in netplan.io (Ubuntu):
status: New → Fix Released
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in maas (Ubuntu):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers