Configuration files for networkd are created when NetworkManager is the default renderer

[Alfonso Sanchez-Beato]

This is happening in a UC image created with a gadget that disables console-conf:

$ ubuntu-image snap --snap=network-manager=22 --snap pc_22.snap

The snaps are:

$ snap list
Name Version Rev Tracking Publisher Notes
core22 20240321 1344 latest/edge canonical✓ base
network-manager 1.36.6-9 876 22/stable canonical✓ -
pc 22-0.3 x1 - - gadget
pc-kernel 5.15.0-102.112.1+1 1731 22/beta canonical✓ kernel
snapd 2.62+git2017.g1afc063e 21490 latest/edge canonical✓ snapd

On first boot, the content in /etc/netplan is:

ubuntu@ubuntu:~$ cat /etc/netplan/00-default-nm-renderer.yaml
network:
  renderer: NetworkManager
ubuntu@ubuntu:~$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2

But we have a configuration file for systemd-networkd that should not be there:

ubuntu@ubuntu:~$ cat /run/systemd/network/10-netplan-ens3.link
[Match]
PermanentMACAddress=52:54:00:12:34:56

[Link]
Name=ens3
WakeOnLan=off

ubuntu@ubuntu:~$ networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 ens3 ether routable configured

While having to:

ubuntu@ubuntu:~$ sudo cat /run/NetworkManager/system-connections/netplan-ens3.nmconnection
[connection]
id=netplan-ens3
type=ethernet
interface-name=ens3

[ethernet]
wake-on-lan=0

[ipv4]
method=auto

[ipv6]
method=ignore

ubuntu@ubuntu:~$ nmcli c
NAME UUID TYPE DEVICE
netplan-ens3 bec3d02a-c9e5-3283-92ab-ee43a4246c85 ethernet ens3

ubuntu@ubuntu:~$ nmcli d
DEVICE TYPE STATE CONNECTION
ens3 ethernet connected netplan-ens3
lo loopback unmanaged --

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Interestingly, networkd seems to take control of the interface after 6-8 minutes from boot. Initially, the state of ens3 is "unconfigured". State of the link before taking control is:

$ cat /run/systemd/netif/links/2
ADMIN_STATE=unmanaged
OPER_STATE=routable
CARRIER_STATE=carrier
ADDRESS_STATE=routable
IPV4_ADDRESS_STATE=routable
IPV6_ADDRESS_STATE=degraded
ONLINE_STATE=online
REQUIRED_FOR_ONLINE=yes
REQUIRED_OPER_STATE_FOR_ONLINE=degraded
REQUIRED_FAMILY_FOR_ONLINE=any
ACTIVATION_POLICY=up
NETWORK_FILE=/run/systemd/network/10-netplan-ens3.network
DNS=10.0.2.3
NTP=
SIP=
DOMAINS=
ROUTE_DOMAINS=
LLMNR=yes
MDNS=no
DHCP_LEASE=/run/systemd/netif/leases/2
DHCP6_CLIENT_DUID=DUID-EN/Vendor:0000ab11c03a37ae49cc0a3d0000

After control is taken, it is the same except ADMIN_STATE=configured.

After reboot, if forcing a restart of networkd:

$ cat /run/systemd/netif/links/2
# This is private data. Do not parse.
ADMIN_STATE=unmanaged
OPER_STATE=routable
CARRIER_STATE=carrier
ADDRESS_STATE=routable
IPV4_ADDRESS_STATE=routable
IPV6_ADDRESS_STATE=degraded

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

On first boot /run/systemd/netif/leases/ is not empty, while for later reboots it is as networkd has not ever controlled ens3.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

The issue most probably comes from systemd-networkd not being restarted after the network-manager snap does a netplan apply via dbus call. As in the end we have

/run/systemd/network/10-netplan-ens3.link

but not

/run/systemd/network/10-netplan-ens3.network

while the latter is still referenced in the internal state of networkd reflected by /run/systemd/netif/links/2.

[Lukas Märdian (slyon)]

Right.. This seems to be a left-over from before the NetworkManager snap takes control, using "renderer: NetworkManager".

/run/systemd/network/10-netplan-ens3.link Is a configuration file for systemd-udev, not systemd-networkd, and takes care of renaming the interface "set-name: ens3". So I think it should be fine being there. But there shouldn't be any .network file and the networkctl state should be "unmanaged".

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

https://github.com/canonical/netplan/pull/449 fixes the issues with networkd ignoring the configuration.

[Lukas Märdian (slyon)]

> alfonsosanchezbeato: to be clear, the networkctl reload/reconfigure commands seem to work, but after 6-8 minutes the state goes back to the one expressed by the old .network files
> alfonsosanchezbeato: you can even force the revert to happen sooner with sudo udevadm trigger, with that it tries to get control back too

[Lukas Märdian (slyon)]

Alfonso, do you have any debug-logs for sytemd-networkd and udev to see what happens when the interface enters the "failed" state?

[Lukas Märdian (slyon)]

I think I can reporudce this on classic...

lxc launch --vm ubuntu-daily:jammy nd-reload
lxc shell nd-reload
root@nd-reload:~# netplan get
network:
  version: 2
  ethernets:
    enp5s0:
      dhcp4: true
root@nd-reload:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 enp5s0 ether routable configured

2 links listed.
root@nd-reload:~# snap install network-manager --channel=22/stable
root@nd-reload:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 enp5s0 ether routable unmanaged

2 links listed.
root@nd-reload:~# udevadm trigger
root@nd-reload:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 enp5s0 ether routable failed
root@nd-reload:~# networkctl reload
root@nd-reload:~# networkctl reconfigure enp5s0
root@nd-reload:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 enp5s0 ether carrier unmanaged

2 links listed.
root@nd-reload:~# udevadm trigger
root@nd-reload:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 enp5s0 ether carrier failed

2 links listed.

After calling "systemctl restart systemd-networkd.service" once, the issue seems to be gone.

I do not yet fully understand what the failure condition is, though?
"networkctl" show a "failed" link. But NetworkManager is managing it fine and the network is reachable.

[Lukas Märdian (slyon)]

Here I produced some networkd debug logs on a UC22 system.

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

To be clear, that carrier is set to "failed" when retriggering uevents is not the main problem. The main problem is that after 5-10 minutes, networkd silently takes control again of the ethernet interface. The retriggering of uevents and the reaction on the networkd side is just an indication that it still cares about the interface.

Maybe a better way to check if the issue is around without having to wait for minutes would be to look at /run/systemd/netif/links/* and check if the content still references the removed .network files.