systemd-networkd not renaming devices on boot

Bug #1770082 reported by Daniel Axtens on 2018-05-09
44
This bug affects 5 people
Affects Status Importance Assigned to Milestone
netplan
Undecided
Unassigned
cloud-init (Ubuntu)
Undecided
Unassigned
netplan.io (Ubuntu)
Undecided
Mathieu Trudel-Lapierre
Bionic
Undecided
Unassigned
nplan (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

[Impact]
Systems relying on renaming network interfaces at boot and when 'netplan apply' is run.

[Test case]
- Write a new netplan YAML (adjusting for current system as necessary):
network:
    version: 2
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: 52:54:00:de:bd:f6
            set-name: myif0

- Run 'netplan apply'
- Verify that the device is correctly renamed to 'myif0'.
- Reboot.
- Make sure the device is correctly renamed to 'myif0'.

[Regression potential]
Changes in rename logic to add udev rules may otherwise impact applying different settings to the network interfaces. Changes in settings on network interfaces, missing parameters (especially on bonds, bridges) should be investigated as potential regressions. Other failures to apply network settings might also happen if there's a race between applying renames via the udev rules, and using the new names to apply configuration changes to the interfaces.

=== systemd issue ===

Renaming devices doesn't seem to work.

If I disable all other network configuration and create /etc/systemd/network/10-network.link with:

[Match]
MACAddress=52:54:00:c1:c9:bb

[Link]
Name=myiface3

I expect this to cause the device with that MAC address to be renamed to myiface3. However, when I reboot, I instead see:

$ ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:c1:c9:bb brd ff:ff:ff:ff:ff:ff

The device is not renamed.

This link file is pretty much identical to Example 2 in https://www.freedesktop.org/software/systemd/man/systemd.link.html.

The renaming does work if I boot with net.ifnames=0, and oddly, it also works if I unbind the device and rebind it as netplan apply does. No setting of NamePolicy seems to help.

=== Original Bug ==

'set-name:' doesn't change the name of a network interface on boot, it only works when you do netplan apply.

Say I take this 50-cloud-init.yaml file:

# This file is generated from information provided by
# the datasource. Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: 52:54:00:de:bd:f6
            set-name: ens3

Say I change set-name to 'myiface3' and reboot. I expect that the device will be called myiface3 and brought up fine with dhcp. However, instead I see:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:de:bd:f6 brd ff:ff:ff:ff:ff:ff

The name has not been changed, and the device has not been brought up.

If I run netplan apply however, I see the following:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: myiface3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:de:bd:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.151/24 brd 192.168.122.255 scope global dynamic myiface3
       valid_lft 3575sec preferred_lft 3575sec
    inet6 fe80::5054:ff:fede:bdf6/64 scope link
       valid_lft forever preferred_lft forever

So names are successfully changed with netplan apply.

This seems to be some udev-related timing or priority issue that I'm still trying to hunt down.

This breaks some forms of migration in certain cloud environments.

Daniel Axtens (daxtens) on 2018-05-09
Changed in netplan:
status: New → Confirmed
Daniel Axtens (daxtens) on 2018-05-09
description: updated
Daniel Axtens (daxtens) on 2018-05-09
description: updated
description: updated
description: updated
description: updated
tags: added: bionic
Daniel Axtens (daxtens) on 2018-05-09
description: updated
Daniel Axtens (daxtens) on 2018-05-09
summary: - set-name doesn't work on boot, only with netplan apply
+ systemd-networkd not renaming devices on boot
Daniel Axtens (daxtens) on 2018-05-09
description: updated

Works for me. I've been using a config like this for a while, the system sets the interface name at boot as expected:

network:
  version: 2
  renderer: networkd
  ethernets:
    mainif:
      match:
        macaddress: "b8:27:eb:b6:35:86"
      set-name: mainif
      addresses: [ "10.3.0.5/23" ]
      gateway4: 10.3.0.1
      [...]

The difference here is that this is hardware rather than a VM. Are we sure that the network interface shows up sufficiently early, and that the mac can be matched at the time?

Changed in netplan:
status: Confirmed → Incomplete
Daniel Axtens (daxtens) wrote :

Both the original cloudy report and my test use a VM. The issue seems to happen for both virtio and emulated e1000 network hardware.

What happens differently with a VM?

Ryan Harper (raharper) wrote :

Are you providing network config to cloud-init? Otherwise, cloud-init is going to generate a fallback network config like your original, dhcp on one of the interfaces.

If you want to persist network changes, you need to add them to a separate netplan yaml config, or provide an updated network-config to cloud-init; otherwise it's going to regenerate the same configuration which will clobber a local change.

Ryan: I removed /etc/netplan/50-cloud-init.yaml and rebooted repeatedly and
I've never seen it regenerated.

I also don't see anything in /run/netplan or /run/systemd/network that has
been autogenerated.

I haven't touched anything else generated by cloud-init. When would
cloud-init regenerate a network config file?

Ryan Harper (raharper) wrote :

On Wed, May 9, 2018 at 9:13 AM, Daniel Axtens
<email address hidden> wrote:
> Ryan: I removed /etc/netplan/50-cloud-init.yaml and rebooted repeatedly and
> I've never seen it regenerated.

Ah, right, it doesn't regenerate unless you change instance-ids.

>
> I also don't see anything in /run/netplan or /run/systemd/network that has
> been autogenerated.
>
> I haven't touched anything else generated by cloud-init. When would
> cloud-init regenerate a network config file?

Nothing, you're right. You can make a local change and reboot but typically
for cloud instances you'd want to have it set during the first boot.

So your scenario is:

1) deploy instance
2) log in and modify /etc/netplan/50-cloud-init.yaml to add a set-name:
3) reboot?

And the result is that you don't see a name change?

Yes, I think this is the udev scenario where it won't change a device
name whose 'name_assign_type' is either 3 or 4;

The driver unplug code in netplan is designed to workaround this
design issue with udev itself.

Can you confirm the driver for the interface you're renaming? ethtool
-i <iface name>

>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Ryan Harper (raharper) wrote :

Investigating, it appears that the interfaces are getting renamed by udev in the initramfs, which pushes their name_assign_type value to 4, and udev refuses to rename an interface if the name_assign_type value is 4.

So, I can make this scenario work if I do:

4. sudo update-initramfs -u
5. reboot

After rebooting, the interface is correct.

I'll add a systemd task here in case we want to poke at allowing renames even if assign-type is 4.

Daniel Axtens (daxtens) wrote :

Ryan, I can't get that to work on my systems. What is it that update-initramfs is supposed to change? I don't see any files in my initramfs that are generated or read by netplan. Neither do I see netplan itself in the initramfs!

Mathieu, I thought netplan was supposed to be initramfs friendly by virtue of being in C. Is it supposed to be in the initramfs?

For both of you, how are you getting through initrd without your device being given a generic udev name (like ens3 or enp1s0)? On both my physical and virtual machines, my various network cards all get renamed in initrd, well before netplan is run.

[PS. I know the scenario as described sounds far-fetched - the original way this came up was a cloud setup where you can change the type of a network interface but keep the same MAC. The different type leads to a different udev name, which is what revealed that set-name: wasn't taking effect.]

Ryan Harper (raharper) wrote :

On Fri, May 11, 2018 at 3:25 AM, Daniel Axtens
<email address hidden> wrote:
> Ryan, I can't get that to work on my systems. What is it that update-
> initramfs is supposed to change? I don't see any files in my initramfs
> that are generated or read by netplan. Neither do I see netplan itself
> in the initramfs!

The .link files end up getting pulled into the initramfs, where
systemd-udev will run and apply the
names there. When systemd-udev does a rename, the kernel marks an
interface 'name_assign_type' to a value of 4
and systemd-udev will refuse to further rename a device, despite
having a rule (.link file) to do so.

Netplan works around this in 'apply' by unplugging the driver, which
resets the state of the 'name_assign_type' value
in the kernel.

>
> Mathieu, I thought netplan was supposed to be initramfs friendly by
> virtue of being in C. Is it supposed to be in the initramfs?

The generator is, the cli is not.

>
> For both of you, how are you getting through initrd without your device
> being given a generic udev name (like ens3 or enp1s0)? On both my
> physical and virtual machines, my various network cards all get renamed
> in initrd, well before netplan is run.

We're not; udev runs in the initramfs and triggers renames there.

>
> [PS. I know the scenario as described sounds far-fetched - the original
> way this came up was a cloud setup where you can change the type of a
> network interface but keep the same MAC. The different type leads to a
> different udev name, which is what revealed that set-name: wasn't taking
> effect.]

Cloud scenarios are slightly different when cloud-init is involved;
Cloud-init itself will handle device renames if
they are needed. Typically cloud-init just re-uses the name the
interface got from udev when it renders it's netplan config during
boot.

For non-cloud scenarios (or if you attempt to use set-name in a cloud
instances without launching a new instance) we're seeing this
situation where we are creating new .link files but they need to be
present in the initramfs otherwise when udevd runs, it will set the
name
to the predictable name and then ignore any of the .link files on the rootfs.

This really needs fixing in udev; I don't think there's a good reason
to have udev refuse to rename an interface.

>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Daniel Axtens (daxtens) wrote :

Ok, so the bit I'm stuck on is how the link files and the netplan generator are getting pulled into the initramfs then.

ubuntu@btest:~$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.15.0-20-generic

ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep \\.link
lib/systemd/network/99-default.link
ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep netplan
ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep generate
ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep system-generators

As you can see there's no generator and no link files in my initramfs - by what mechanism is it supposed to work? What package/script/tool is supposed to pull the link files in?

Ryan Harper (raharper) wrote :

I'm just verifying that on my end; note that the 99-default.link is
going to be sufficient to trigger a rename from eth0 to ens3; which
then prevents any rename once we mount root.

Here's the trigger:

% /usr/share/initramfs-tools/hooks/udev
# copy .link files containing interface naming definitions
mkdir -p $DESTDIR/lib/systemd/network/
find /lib/systemd/network -name '*.link' -execdir cp -pt
$DESTDIR/lib/systemd/network/ '{}' +
if [ -d /etc/systemd/network ]; then
  find /etc/systemd/network -name '*.link' -execdir cp -pt
$DESTDIR/lib/systemd/network/ '{}' +
fi

If you create a .link file in /etc/systemd/network/ lower than 99-*
then those link rules will apply first.

On Fri, May 11, 2018 at 9:24 AM, Daniel Axtens
<email address hidden> wrote:
> Ok, so the bit I'm stuck on is how the link files and the netplan
> generator are getting pulled into the initramfs then.
>
> ubuntu@btest:~$ sudo update-initramfs -u
> update-initramfs: Generating /boot/initrd.img-4.15.0-20-generic
>
> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep \\.link
> lib/systemd/network/99-default.link
> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep netplan
> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep generate
> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep system-generators
>
> As you can see there's no generator and no link files in my initramfs -
> by what mechanism is it supposed to work? What package/script/tool is
> supposed to pull the link files in?
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Ryan Harper (raharper) wrote :

root@rharper-b1:~# vi /etc/netplan/50-cloud-init.yaml
# changed the set-name: to eth_lan
root@rharper-b1:~# netplan generate
root@rharper-b1:~# cp /run/systemd/network/10-netplan-ens3.link
/etc/systemd/network/

# just the default link file in the initramfs
root@rharper-b1:~# lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep \.link
lib/systemd/network/99-default.link
lib/modules/4.15.0-20-generic/kernel/net/core/devlink.ko
lib/udev/rules.d/80-net-setup-link.rules
bin/readlink

root@rharper-b1:~# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-4.15.0-20-generic

# now we see the local .link file from /etc copied in.
root@rharper-b1:~# lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep \.link
lib/modules/4.15.0-20-generic/kernel/net/core/devlink.ko
lib/systemd/network/99-default.link
lib/systemd/network/10-netplan-ens3.link
lib/udev/rules.d/80-net-setup-link.rules
bin/readlink

On Fri, May 11, 2018 at 9:35 AM, Ryan Harper <email address hidden> wrote:
> I'm just verifying that on my end; note that the 99-default.link is
> going to be sufficient to trigger a rename from eth0 to ens3; which
> then prevents any rename once we mount root.
>
> Here's the trigger:
>
> % /usr/share/initramfs-tools/hooks/udev
> # copy .link files containing interface naming definitions
> mkdir -p $DESTDIR/lib/systemd/network/
> find /lib/systemd/network -name '*.link' -execdir cp -pt
> $DESTDIR/lib/systemd/network/ '{}' +
> if [ -d /etc/systemd/network ]; then
> find /etc/systemd/network -name '*.link' -execdir cp -pt
> $DESTDIR/lib/systemd/network/ '{}' +
> fi
>
> If you create a .link file in /etc/systemd/network/ lower than 99-*
> then those link rules will apply first.
>
>
> On Fri, May 11, 2018 at 9:24 AM, Daniel Axtens
> <email address hidden> wrote:
>> Ok, so the bit I'm stuck on is how the link files and the netplan
>> generator are getting pulled into the initramfs then.
>>
>> ubuntu@btest:~$ sudo update-initramfs -u
>> update-initramfs: Generating /boot/initrd.img-4.15.0-20-generic
>>
>> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep \\.link
>> lib/systemd/network/99-default.link
>> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep netplan
>> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep generate
>> ubuntu@btest:~$ lsinitramfs /boot/initrd.img-4.15.0-20-generic | grep system-generators
>>
>> As you can see there's no generator and no link files in my initramfs -
>> by what mechanism is it supposed to work? What package/script/tool is
>> supposed to pull the link files in?
>>
>> --
>> You received this bug notification because you are subscribed to
>> netplan.
>> Matching subscriptions: netplan
>> https://bugs.launchpad.net/bugs/1770082
>>
>> Title:
>> systemd-networkd not renaming devices on boot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Daniel Axtens (daxtens) wrote :

Right. Yes, I can replicate that, thanks heaps!

So to summarise, you can make the rename take effect on boot if you:
 1) copy the files from /run/systemd/network -> /etc/systemd/network
 2) then update-initramfs -u

This seems pretty far outside of the way that netplan is supposed to work -
indeed using /etc/systemd/ is basically bypassing netplan. So we still have
the issue that just changing 'set-name' in /etc/netplan/*.yaml doesn't work
as expected. What should we change so that set-name works as expected?

I see a few options:

 0) Document that set-name is fragile and stop relying on it. This is
actually really easy to do and I have a netplan patch drafted already. The
reason that we have no network connectivity when set-name fails is that the
network file netplan creates maches on *both* the new name and the mac
address. We can just drop the new name from the [Match] section of the
relevant .network file and match only on the provided mac address (or
whatever else was used in the netplan match stanza). This means that
regardless of the interface name it's brought up and networking works.

 1) Make the initramfs hook for udev copy the files from /run (as well as
from /lib and /etc) into the initial ramdisk. This is probably my
preference; we can run netplan generate beforehand to make sure we get a
clean copy, and then document that update-initramfs is required to get
stuff to stick.

 2) Allow udev to re-rename interfaces. I don't know if this would be
acceptable to systemd upstream?

 3) Something else?

Regards,
Daniel

Ryan Harper (raharper) wrote :

On Fri, May 11, 2018 at 10:18 AM, Daniel Axtens
<email address hidden> wrote:
> Right. Yes, I can replicate that, thanks heaps!

Great!

>
> So to summarise, you can make the rename take effect on boot if you:
> 1) copy the files from /run/systemd/network -> /etc/systemd/network
> 2) then update-initramfs -u
>
> This seems pretty far outside of the way that netplan is supposed to work -
> indeed using /etc/systemd/ is basically bypassing netplan. So we still have
> the issue that just changing 'set-name' in /etc/netplan/*.yaml doesn't work
> as expected. What should we change so that set-name works as expected?
>
> I see a few options:
>
> 0) Document that set-name is fragile and stop relying on it. This is
> actually really easy to do and I have a netplan patch drafted already. The
> reason that we have no network connectivity when set-name fails is that the
> network file netplan creates maches on *both* the new name and the mac
> address. We can just drop the new name from the [Match] section of the
> relevant .network file and match only on the provided mac address (or
> whatever else was used in the netplan match stanza). This means that
> regardless of the interface name it's brought up and networking works.

IIUC, we want to stop include the Name= value in the .network config since the
name is flaky; we may want to hold off on that as I think we want the naming to
be solid. You've outlined at least one way below. We should make
set-name reliable
and we can do that, even within netplan itself.

>
> 1) Make the initramfs hook for udev copy the files from /run (as well as
> from /lib and /etc) into the initial ramdisk. This is probably my
> preference; we can run netplan generate beforehand to make sure we get a
> clean copy, and then document that update-initramfs is required to get
> stuff to stick.

Yes, I think that's also reasonable, it's a new location where .link
files may be present.

>
> 2) Allow udev to re-rename interfaces. I don't know if this would be
> acceptable to systemd upstream?
>
> 3) Something else?

Netplan can do what cloud-init does and check that if the current name
of an interface doesn't match
the set-name value, to ip link set name on it.

>
> Regards,
> Daniel
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Daniel Axtens (daxtens) wrote :

Ok, how does this look?

The attachment "link-files-in-run.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Daniel Axtens (daxtens) wrote :

This is not a full solution. In Xenial this renaming works even if initramfs has *not* been updated and there is no 70-persistent-net.rules file in the initial ramdisk. I'm still figuring out why this is, but it means that even if my patch were applied, there would be a regression in bionic vs xenial.

Here's a snippet from dmesg showing the same device being renamed twice, this is a xenial guest. It shows the device going from the kernel name (eth0) to the udev slot based name (ens16), to the name specified in the udev rules file that is *not* present in initrd.

ubuntu@xt2:~$ dmesg|grep renamed
[ 2.428015] virtio_net virtio3 ens16: renamed from eth0
[ 5.317990] virtio_net virtio3 ens3: renamed from ens16

Ryan Harper (raharper) wrote :

Can you provide the journalctl -b output for the boot that does this
rename? Is the reproduce the same, can you provide your steps and
I'll replicate.

Xenial does not enable netplan by default so we have a different path
there at least.

On Mon, May 14, 2018 at 9:58 PM, Daniel Axtens
<email address hidden> wrote:
> This is not a full solution. In Xenial this renaming works even if
> initramfs has *not* been updated and there is no 70-persistent-net.rules
> file in the initial ramdisk. I'm still figuring out why this is, but it
> means that even if my patch were applied, there would be a regression in
> bionic vs xenial.
>
> Here's a snippet from dmesg showing the same device being renamed twice,
> this is a xenial guest. It shows the device going from the kernel name
> (eth0) to the udev slot based name (ens16), to the name specified in the
> udev rules file that is *not* present in initrd.
>
> ubuntu@xt2:~$ dmesg|grep renamed
> [ 2.428015] virtio_net virtio3 ens16: renamed from eth0
> [ 5.317990] virtio_net virtio3 ens3: renamed from ens16
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Daniel Axtens (daxtens) wrote :

Hi Ryan,

[Journal Output]
Attached.

[Reproducer]
uvt-kvm create xenial-test release=xenial arch=amd64
virsh edit xenial-test # change network interface pci slot: s/0x03/0x10/
virsh destroy xenial-test
virsh start xenial-test
uvt-kvm ssh xenial-test
dmesg|grep rename
[ 2.790623] virtio_net virtio3 ens16: renamed from eth0
[ 6.048520] virtio_net virtio3 ens3: renamed from ens16

[Analysis]
I've been working on this a lot, and I think I have the cause of the difference.

In udev-events.c, udev_execute_rules will _forcibly_ rename a device with via a netlink message if there is a matching rule that sets a name. Under Xenial, there *is* a matching rule, in 70-persistent-net.rules, so this forces a rename. This rename will occur even if the device already has a name, and therefore even if the rules file isn't in the initramfs.

Under Bionic, this rules file doesn't exist, there is a .link file instead. Unfortunately, a name in a .link file will only take effect if the device hasn't been renamed - because of the renaming in initrd, this means that a link file that is not present in the initrd will never be able to cause a rename.

[Solutions]
There are a couple of ways we could fix this that come to mind:

 - make netplan generate a file in /run/udev/rules.d for each device
 - make systemd rename devices from a link file even if they've been renamed

My preference is the first, but I'm open to anything we can get upstream.

Thanks again.

Regards,
Daniel

Ryan Harper (raharper) wrote :

Note, that uvt-kvm is going to use cloud-init; how are you making
sure that cloud-init isn't doing the rename itself?
Xenial by default doesn't use netplan; it still uses eni; the network
configuration generation in cloud-init using eni
does create a 70-persistent-net.rules file that handles renames on
subsequent reboots; in a netplan world (bionic)
the .link file is meant to be the equivalent of a .rules file.

I've not attempted to determine if systemd-udev in bionic would
respect renames from .rules file; it certainly seems
odd to have .rules files allow renames independent of name_assign_type
value where .link files do.

On Tue, May 15, 2018 at 12:29 PM, Daniel Axtens
<email address hidden> wrote:
> Hi Ryan,
>
> [Journal Output]
> Attached.
>
> [Reproducer]
> uvt-kvm create xenial-test release=xenial arch=amd64
> virsh edit xenial-test # change network interface pci slot: s/0x03/0x10/
> virsh destroy xenial-test
> virsh start xenial-test
> uvt-kvm ssh xenial-test
> dmesg|grep rename
> [ 2.790623] virtio_net virtio3 ens16: renamed from eth0
> [ 6.048520] virtio_net virtio3 ens3: renamed from ens16
>
> [Analysis]
> I've been working on this a lot, and I think I have the cause of the difference.
>
> In udev-events.c, udev_execute_rules will _forcibly_ rename a device
> with via a netlink message if there is a matching rule that sets a name.
> Under Xenial, there *is* a matching rule, in 70-persistent-net.rules, so
> this forces a rename. This rename will occur even if the device already
> has a name, and therefore even if the rules file isn't in the initramfs.
>
> Under Bionic, this rules file doesn't exist, there is a .link file
> instead. Unfortunately, a name in a .link file will only take effect if
> the device hasn't been renamed - because of the renaming in initrd, this
> means that a link file that is not present in the initrd will never be
> able to cause a rename.
>
> [Solutions]
> There are a couple of ways we could fix this that come to mind:
>
> - make netplan generate a file in /run/udev/rules.d for each device
> - make systemd rename devices from a link file even if they've been renamed
>
> My preference is the first, but I'm open to anything we can get
> upstream.
>
> Thanks again.
>
> Regards,
> Daniel
>
> ** Attachment added: "journalctl -b output on Xenial VM with multiple renames"
> https://bugs.launchpad.net/netplan/+bug/1770082/+attachment/5139894/+files/journalctl-b.txt
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Daniel Axtens (daxtens) wrote :

> Note, that uvt-kvm is going to use cloud-init; how are you making
> sure that cloud-init isn't doing the rename itself?

I instrumented the kernel. I added a call to dump_stack() in the function that does interface renaming: dev_change_name() in net/core/dev.c. That showed that the process that sent the netlink message was systemd-udevd:

[ 1.007200] virtio_net virtio3 ens16: renamed from eth0
[ 1.008453] CPU: 0 PID: 124 Comm: systemd-udevd Not tainted 4.4.59+ #1
[ 1.009887] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 1.011938] 0000000000000286 00000000d38795d9 ffff88001ed3b970 ffffffff813f3f63
[ 1.012435] ffff88001f2d0000 0000000000000000 ffff88001ed3b9d8 ffffffff8170669b
[ 1.012435] ffff88001ed3b9c8 ffff88001ed3b9a0 ffffffff812255af 0000000030687465
[ 1.012435] Call Trace:
[ 1.012435] [<ffffffff813f3f63>] dump_stack+0x63/0x90
[ 1.012435] [<ffffffff8170669b>] dev_change_name+0x25b/0x2e0
<call chain back to syscall snipped>

...

[ 2.336114] virtio_net virtio3 ens3: renamed from ens16
[ 2.336121] CPU: 0 PID: 454 Comm: systemd-udevd Not tainted 4.4.59+ #1
[ 2.336122] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 2.336124] 0000000000000286 0000000027d63c85 ffff88001c59b970 ffffffff813f3f63
[ 2.336126] ffff88001f2d0000 0000000000000000 ffff88001c59b9d8 ffffffff8170669b
[ 2.336128] 0000000000000001 ffff88001c59b9b0 ffffffff810b3f1c 0000003631736e65
[ 2.336130] Call Trace:
[ 2.336135] [<ffffffff813f3f63>] dump_stack+0x63/0x90
[ 2.336138] [<ffffffff8170669b>] dev_change_name+0x25b/0x2e0
<call chain back to syscall snipped>

Notice in both cases: "Comm: systemd-udevd". Also notice the change in PID, suggesting that the first one is in initrd and the second one is not.

I then grepped the source code for netlink stuff and quickly narrowed in on rename_if in udev-events.c.

> Xenial by default doesn't use netplan; it still uses eni; the network
> configuration generation in cloud-init using eni
> does create a 70-persistent-net.rules file that handles renames on
> subsequent reboots; in a netplan world (bionic)
> the .link file is meant to be the equivalent of a .rules file.

I am aware of this. (I've hacked on netplan quite a bit.)

The issue is that a .link file does not seem to be functionally equivalent to a .rules file. I don't know if this difference is deliberate or an oversight. I will open an issue upstream and ask.

> I've not attempted to determine if systemd-udev in bionic would
> respect renames from .rules file; it certainly seems
> odd to have .rules files allow renames independent of name_assign_type
> value where .link files do.

The end-user tested with a .rules file on Bionic and reported that it worked. I have also just verified that it does respect renames from a rules file regardless of name_assign_type. (The codepaths are unchanged between systemd-229 and systemd-237.)

I was struggling with this for quite some time and played around using netplan.io and not using it. I figured out that you need to run "update-initramfs -u" to make settings persist reboots in "/etc/systemd/network/" when not using netplan.io.

But netplan.io saves the configuration in "/run/systemd/network/". Here the update of initramfs does not make any difference and on boot the network set-up fails to rename the interfaces as configured with all consequences for the system.

To me it appears that "update-initramfs" needs to include the generated configuration from netplan.io to get the network configuration correct on reboot.

The following workaround does the job for me after every configuration change in netplan.io:

1. netplan apply
2. rm /etc/systemd/network/10-netplan-*.link
3. cp /run/systemd/network/10-netplan-*.link /etc/systemd/network/
4. update-initramfs -u

Now the generated link files are found by update-initrd. Never the less there should be a consistent solution.

Mike Jonkmans (mjo) wrote :

Things seem to work when i change the NamePolicy in the initrd file /lib/systemd/network/99-default.link to
`NamePolicy=kernel`

Leaving out the other NamePolicy-options, prevents renaming in the initrd fase.
Which allows later renames from .link files/netplan.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Ryan Harper (raharper) wrote :

We really only want to be allowed to rename interfaces that have requested it.
This means other interfaces which do not have a set-name directive
will have the kernel name.

On Thu, May 24, 2018 at 6:56 AM, Mike Jonkmans
<email address hidden> wrote:
> Things seem to work when i change the NamePolicy in the initrd file /lib/systemd/network/99-default.link to
> `NamePolicy=kernel`
>
> Leaving out the other NamePolicy-options, prevents renaming in the initrd fase.
> Which allows later renames from .link files/netplan.
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Mike Jonkmans (mjo) wrote :

As discussed on https://github.com/systemd/systemd/issues/9006

The initrd (!) should have NamePolicy=kernel in /lib/systemd/network/99-default.link

Ryan Harper (raharper) wrote :

That's not the final conclusion; the issue is still being discussed upstream; in particular as to why only the interface name cannot be modified at runtime by .link files.

Changed in netplan:
status: Incomplete → Confirmed
Changed in cloud-init (Ubuntu):
status: New → Confirmed

There are a couple of pieces in play here.

One aspect is that we don't really want to write the .link files for systemd-networkd to /lib or get anything from /run into the initrd -- that defeats the purpose of netplan's config being dynamic.

The second aspect is that depending on how the systems are deployed (MAAS? cloud images? openstack? VM? hardware?) changes the behavior a bit. Some USB-based network interfaces are simply "unaffected" and rename just fine at boot. Other systems may be renaming the interfaces, but seeing cloud-init re-change the name *back* to the previous value (this has been observed for MAAS-deployed systems). This is why I added cloud-init to affected packages -- cloud-init should not be second-guessing the network layer and attempting to do renames / to run udevadm, except *maybe* at first boot/when the instance data is created. I believe it should just leave it alone completely, and let netplan/systemd/NM deal with the interfaces themselves (if anything needs to be poked, it's arguably a bug in the backend).

Another issue is that systemd is also second-guessing configuration. If we set things to be renamed, we do want them to be renamed. Now, this is a bit more up to discussion upstream, but I do think systemd should always rename if configuration tells it to (via .link files or udev rules -- .link files are "cleaner", easier to write). Users renaming their interfaces manually via 'ip link dev X name Y' are doing so on their own, and the change would not be persistent across a reboot -- renaming in configuration is expected to work normally.

My recommended course of action for cosmic:

 - drop udevadm (net_setup_link) call from cloud-init
 - drop set-name "renaming" from cloud-init / maas
 - drop replug code in netplan; replace with proper .link code, possibly call to net_setup_link.
 - maybe write udev rule for renaming in netplan (think belt and suspenders)
 - make sure .link files are written correctly by netplan for renaming
 - patch systemd to not second-guess renaming from .link files

My recommended course of action for SRUs:

 - write udev .rules files from netplan to enforce renaming
 - drop udevadm (net_setup_link) call from cloud-init

The above should be sufficient and non-intrusive enough for SRU. The tasks for cosmic are additional changes to clean up the behavior for the future.

netplan changes are available in git:

Daniel's patch to write udev rules (SRU material):
https://github.com/CanonicalLtd/netplan/commit/b0c51bfa8ba8b898a9feaed9cd7d8790d147d35d

Daniel's patch + dropping replug code + rework 'netplan apply' (code for cosmic); in progress for upload to cosmic:
https://github.com/CanonicalLtd/netplan/tree/live-rename

I'm not completely sure where the code lives in cloud-init; it looks a bit like what's in:

cloudinit/net/netplan.py

But the code does read as though it should not be running 'udevadm test-builtin net_setup_link'. However, deploying a system with MAAS shows a /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg file, which contains data referring to the network interfaces:

network:
  config:
  - id: ens6
    mac_address: 52:54:00:31:27:2c
    mtu: 1500
    name: ens6
    subnets:
    - address: 10.3.99.13/24
      gateway: 10.3.99.1
      type: static
    type: physical
  version: 1

With that file in place, any netplan config renaming the interface to something other than ens6 will see the interface being renamed *again* back to ens6, when that file is removed, this behavior does not appear.

Ryan Harper (raharper) wrote :

"This is why I added
cloud-init to affected packages -- cloud-init should not be second-
guessing the network layer and attempting to do renames / to run"

There is no second guessing. In the case where we have no network
config, there is no renaming; we accept whatever name is given.

If the config passed to cloud-init includes a name for an interface, then
cloud-init applies that name; this is the MAAS scenario;

Given the unreliable nature of udev w.r.t naming (see this very bug)
The *only* way for cloud-init to ensure that a directive to name an interface
matches the config (also note cloud-init accepts network config in
various formats
not just netplan) is to handle naming if requested directly, precisely
due to this bug.

If we fix systemd-udevd to allow renames of interfaces reliably that helps
most cases where udevd runs. For the remaining cases where udev doesn't run,
containers for example, cloud-init will still need to use iproute2 to
set an interface
name if requested.

On Fri, May 25, 2018 at 8:55 AM, Mathieu Trudel-Lapierre
<email address hidden> wrote:
> netplan changes are available in git:
>
> Daniel's patch to write udev rules (SRU material):
> https://github.com/CanonicalLtd/netplan/commit/b0c51bfa8ba8b898a9feaed9cd7d8790d147d35d
>
> Daniel's patch + dropping replug code + rework 'netplan apply' (code for cosmic); in progress for upload to cosmic:
> https://github.com/CanonicalLtd/netplan/tree/live-rename
>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Ryan Harper (raharper) wrote :

On Fri, May 25, 2018 at 8:48 AM, Mathieu Trudel-Lapierre
<email address hidden> wrote:
> My recommended course of action for cosmic:
>
> - drop udevadm (net_setup_link) call from cloud-init

This will still be needed in the case that we have a network config with names.
Cloud-init runs after udev coldplug, so any .link files would have to already be
present. We're generating a network config post coldplug, but pre networking.

Now, if after we netplan generate, if links could be applied or if
networkd kicks udev
to rename interfaces, that's fine. But it doesn't do that today.

> - drop set-name "renaming" from cloud-init / maas

I've already replied, but this is required for instances where udev
doesn't run but
network config has been provided.

> - drop replug code in netplan; replace with proper .link code, possibly call to net_setup_link.

+1

> - maybe write udev rule for renaming in netplan (think belt and suspenders)

? Instead of the .link files?

> - make sure .link files are written correctly by netplan for renaming
> - patch systemd to not second-guess renaming from .link files

+1

>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Ryan Harper (raharper) wrote :

On Fri, May 25, 2018 at 9:09 AM, Mathieu Trudel-Lapierre
<email address hidden> wrote:
> I'm not completely sure where the code lives in cloud-init; it looks a
> bit like what's in:
>
> cloudinit/net/netplan.py
>
> But the code does read as though it should not be running 'udevadm test-
> builtin net_setup_link'. However, deploying a system with MAAS shows a
> /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg file, which contains
> data referring to the network interfaces:
>
> network:
> config:
> - id: ens6
> mac_address: 52:54:00:31:27:2c
> mtu: 1500
> name: ens6
> subnets:
> - address: 10.3.99.13/24
> gateway: 10.3.99.1
> type: static
> type: physical
> version: 1
>
> With that file in place, any netplan config renaming the interface to
> something other than ens6 will see the interface being renamed *again*
> back to ens6, when that file is removed, this behavior does not appear.

Cloud-init only renders network config once per-instance. If a user
wanted to modify this to something else (and we fix udev to handle
.link file name changes) then there won't be any additional renames.
Even without the change, cloud-init isn't going to call udevadm test-builtin
more than once per-instance.

>
> --
> You received this bug notification because you are subscribed to
> netplan.
> Matching subscriptions: netplan
> https://bugs.launchpad.net/bugs/1770082
>
> Title:
> systemd-networkd not renaming devices on boot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/netplan/+bug/1770082/+subscriptions

Changed in netplan.io (Ubuntu):
status: New → In Progress
assignee: nobody → Mathieu Trudel-Lapierre (cyphermox)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netplan.io - 0.38

---------------
netplan.io (0.38) cosmic; urgency=medium

  * New upstream release:
    - Write udev .rules files to /run/udev/rules.d to enforce interface
      renaming. (LP: #1770082)
    - Don't traceback for 'netplan ip leases' when iface is not managed or
      doesn't DHCP (LP: #1768823)
    - Fix duplicate "/" path separator in error messages (LP: #1771440)
    - Fix incorrect terminal reset in 'netplan try' on Ctrl-C. (LP: #1768798)
    - Updated doc entries: mtu, fix fwmark->mark, cleanup optional.
      (LP: #1768783)
    - Added documentation validation at build.
    - Added configuration example for multi-ip interfaces.
  * tests/integration.py: fix test_eth_and_bridge autopkg test harder.
  * debian/control:
    - Add iproute2 to Depends.
    - Add python3-netifaces to Depends, Build-Depends.

 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 04 Jun 2018 14:45:26 -0400

Changed in netplan.io (Ubuntu):
status: In Progress → Fix Released
Changed in nplan (Ubuntu):
status: New → Fix Released
Nicorac (nicorac) wrote :

Thanks for the update.

Referring to this duplicate bug (https://bugs.launchpad.net/netplan/+bug/1768827), is it going to be backported to 18.04?
Or is there a way to install 0.38 on Ubuntu Server 18.04?

Łukasz Zemczak (sil2100) wrote :

Could this bug be updated with SRU relevant information? Also, it would also be nice if we had the same changes prepared for netplan.io in bionic.

description: updated

Hello Daniel, or anyone else affected,

Accepted nplan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nplan/0.32~16.04.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nplan (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
Daniel Axtens (daxtens) wrote :

Hi Łukasz,

I couldn't find the package in -proposed, but I was able to download it from the link.

I verified that a device was not renamed with 0.32~16.04.5 but was correctly renamed with 0.32~16.04.6, so for Xenial, verification succeeds.

Regards,
Daniel

tags: added: verification-done-xenial
removed: verification-needed-xenial
Steve Langasek (vorlon) wrote :

Hello Daniel, or anyone else affected,

Accepted netplan.io into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/netplan.io/0.36.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in netplan.io (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic
Nicorac (nicorac) wrote :

Partially working here on fully updated Ubuntu 18.04 Server on VirtualBox :(

I've enabled -proposed repo and updated both nplan/bionic-proposed and netplan.ion/bionic-proposed packages.

Now interfaces got renamed, but IPs are not assigned at boot (both DHCP and static ones).
Calling "netplan generate && netplan apply" after boot only sets the static IPs while DHCP still remains unset.

# dpkg -s netplan.io
Package: netplan.io
Status: install ok installed
Priority: important
Section: net
Architecture: amd64
Version: 0.36.3

# cat /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    id0:
      match:
        macaddress: 08:00:27:6b:d8:91
      set-name: eth_static
      addresses: [ 1.2.3.4/16 ]
      gateway4: 5.6.7.8
    id1:
      match:
        macaddress: 08:00:27:23:68:f5
      set-name: eth_dhcp
      dhcp4: true

*** (just after boot) ***
# ifconfig -a
eth_dhcp: flags=4098<BROADCAST,MULTICAST> mtu 1500
        ether 08:00:27:23:68:f5 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth_static: flags=4098<BROADCAST,MULTICAST> mtu 1500
        ether 08:00:27:6b:d8:91 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 2 bytes 78 (78.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 2 bytes 78 (78.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

*** (after "netplan apply") ***
eth_dhcp: flags=4098<BROADCAST,MULTICAST> mtu 1500
        ether 08:00:27:23:68:f5 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

eth_static: flags=4098<BROADCAST,MULTICAST> mtu 1500
        inet 1.2.3.4 netmask 255.255.0.0 broadcast 1.2.255.255
        inet6 fe80::a00:27ff:fe6b:d891 prefixlen 64 scopeid 0x20<link>
        ether 08:00:27:6b:d8:91 txqueuelen 1000 (Ethernet)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
        loop txqueuelen 1000 (Local Loopback)
        RX packets 2 bytes 78 (78.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 2 bytes 78 (78.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

tags: added: verification-failed-bionic
removed: verification-needed-bionic
tags: removed: verification-needed
Daniel Axtens (daxtens) on 2018-07-09
tags: added: verification-done-bionic
removed: verification-failed-bionic
Daniel Axtens (daxtens) wrote :

Hi,

I can verify correct behaviour with renaming, static IPs and DHCP with KVM. I'm about to check some cloud services as well.

Nicorac, are you claiming this is a regression or just an incomplete fix?

If it's just an incomplete fix, I think we should spin that out into another bug and fix it after this SRU goes through.

Regards,
Daniel

Daniel Axtens (daxtens) wrote :

Hi,

I have confirmed that the Bionic package works on the cloud service where the bug was originally observed. With this package, the affected migrations now succeed.

Regards,
Daniel

Nicorac (nicorac) wrote :

@Daniel: no, it's a regression to me because set-name never worked and now DHCP stopped working too.

But I'm not testing the same environment as you, I'm on VirtualBox and this is my original bug report (linked to this one):
https://bugs.launchpad.net/netplan/+bug/1768827

Hi,

I'm a bit confused then.

This is the netplan config you mentioned:

network:
  version: 2
  renderer: networkd
  ethernets:
    id0:
      match:
        macaddress: 08:00:27:6b:d8:91
      set-name: eth_static
      addresses: [ 1.2.3.4/16 ]
      gateway4: 5.6.7.8
    id1:
      match:
        macaddress: 08:00:27:23:68:f5
      set-name: eth_dhcp
      dhcp4: true

With the current version (not the -proposed version), my understanding
is that both devices keep their original names, neither device comes
up at all, and neither device will get an IP, dynamic or static. But
if I understand your comment correctly, you're saying that one of the
devices (the one with mac :f5) comes up and gets a DHCP IP? Is that
right? And if so, what is the device called - eth_dhcp or something
else?

Do you have virtualbox guest additions? I wonder if that's doing
something funky. And just checking you don't have ifupdown installed?

Regards,
Daniel

Nicorac (nicorac) wrote :

Sorry for the confusion.

Current version (0.36.2):
- both devices not renamed after boot (have original enp.. names)
- static and dynamic IPs correctly assigned

Proposed version (0.36.3):
- both devices correctly renamed after boot:
  :91 gets renamed eth_static
  :f5 gets renamed to eth_dhcp
- only eth_static got the configured static address, eth_dhcp not configured (need to call "ifconfig -a" to show it)

No VB Additions, just plain Ubuntu Server 18.04 install followed by "apt update && apt dist-upgrade".

I'd be a bit surprised if there really was a regression due to the change, we're more likely seeing something going down due to the DHCP server being slow to respond, possibly because it already responded before?

What does the 'networkctl' command show immediately after boot?

Let's continue debugging this on your own bug (1768827).

Nicorac (nicorac) wrote :

The behavior is not reproducible, maybe a race condition?
I'm actually not getting IP addres for static interface too :-(

I agree, let's continue there: https://bugs.launchpad.net/netplan/+bug/1768827

Łukasz Zemczak (sil2100) wrote :

Since the new issue is tracked in a separate bug and currently only reproducible by the reporter, I am releasing the current update as is. In case this indeed turns out to be a regression, let's consider fixing it with a subsequent upload.

The verification of the Stable Release Update for netplan.io has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nplan - 0.32~16.04.6

---------------
nplan (0.32~16.04.6) xenial; urgency=medium

  [ Mathieu Trudel-Lapierre ]
  * tests/integration.py: Fix autopkgtests involving bonds/bridges to do proper
    cleanup every time, so later tests don't unnecessarily wait for an
    interface not configured to be up. (LP: #1775097)

  [ Daniel Axtens ]
  * Generate udev rules files to rename devices (LP: #1770082)
    Due to a systemd issue[1], using link files to rename interfaces
    doesn't work as expected. Link files will not rename an interface
    if it was already renamed, and interfaces are renamed in initrd, so
    set-name will often not work as expected when rebooting. However,
    rules files will cause a renaming, even if the interface has been
    renamed in initrd.

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 03 Jul 2018 12:55:11 -0400

Changed in nplan (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package netplan.io - 0.36.3

---------------
netplan.io (0.36.3) bionic; urgency=medium

  * Generate udev rules files to rename devices (LP: #1770082)
    Due to a systemd issue[1], using link files to rename interfaces
    doesn't work as expected. Link files will not rename an interface
    if it was already renamed, and interfaces are renamed in initrd, so
    set-name will often not work as expected when rebooting. However,
    rules files will cause a renaming, even if the interface has been
    renamed in initrd.
  * tests/integration.py: fix test_eth_and_bridge autopkg test harder.
  * tests/integration.py: fix test_mix_bridge_on_bond autopkgtest too.

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 05 Jul 2018 16:54:52 -0400

Changed in netplan.io (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in netplan:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.