network-online.target is reached before netplan-configured interfaces are online

[Lucas Nussbaum]

Hi,

Running Ubuntu 24.04 with netplan.io/netplan-generator 1.0-2ubuntu1.

I use a systemd unit to mount an NFS filesystem. That unit should only be activated when the network is ready. However, it looks like network-online.target is reached before netplan-configured interfaces are up, so the unit triggers too early and sometimes fail (because it cannot resolve the name of the remote server).

I configured netplan manually with:
```
# cat /etc/netplan/01-netcfg.yaml
---
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: true
```

I see the following during boot:

```
Apr 27 23:31:28 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Apr 27 23:31:28 localhost systemd-networkd[1190]: lo: Link UP
Apr 27 23:31:28 localhost systemd-networkd[1190]: lo: Gained carrier
Apr 27 23:31:28 localhost systemd-networkd[1190]: Enumeration completed
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Configuring with /run/systemd/network/10-netplan-eno1.network.
Apr 27 23:31:28 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Apr 27 23:31:28 localhost systemd[1]: Reached target network.target - Network.
Apr 27 23:31:28 localhost systemd[1]: systemd-networkd-wait-online.service - Wait for Network to be Configured was skipped because of an unmet condition check (ConditionPathIsSymbolicLink=/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service).
Apr 27 23:31:28 localhost systemd[1]: Reached target network-online.target - Network is Online.
Apr 27 23:31:28 localhost systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
Apr 27 23:31:28 localhost systemd[1]: Mounting grid5000.mount - /grid5000...
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Link UP
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Gained carrier
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: DHCPv4 address 172.16.66.53/20, gateway 172.16.79.254 acquired from 172.16.79.107
Apr 27 23:31:28 localhost systemd[1]: Starting networkd-dispatcher.service - Dispatcher daemon for systemd-networkd...
Apr 27 23:31:29 gros-53 systemd[1]: Starting rpc-statd.service - NFS status monitor for NFSv2/3 locking....
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Version 2.6.4 starting
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Flags: TI-RPC
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Failed to read /var/lib/nfs/state: Success
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Initializing NSM state
Apr 27 23:31:29 gros-53 systemd[1]: Started rpc-statd.service - NFS status monitor for NFSv2/3 locking..
Apr 27 23:31:29 gros-53 mount[1441]: Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /usr/lib/systemd/system/rpc-statd.service.
Apr 27 23:31:29 gros-53 systemd[1]: Reloading requested from client PID 1441 ('systemctl') (unit grid5000.mount)...
Apr 27 23:31:29 gros-53 systemd[1]: Reloading...
Apr 27 23:31:30 gros-53 systemd[1]: Reloading finished in 194 ms.
Apr 27 23:31:30 gros-53 systemd[1]: Mounted grid5000.mount - /grid5000.
Apr 27 23:31:30 gros-53 nfsrahead[1497]: setting /grid5000 readahead to 128

The key message here seems to be:
Apr 27 23:31:28 localhost systemd[1]: systemd-networkd-wait-online.service - Wait for Network to be Configured was skipped because of an unmet condition check (ConditionPathIsSymbolicLink=/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service).
Why doesn't netplan generate that symlink? It seems it does generate it, but only later, when systemd reloads at 23:31:29 (according to 'stat' on the symlink).

It worked fine with Ubuntu 22.04:
"systemd[1]: Reached target Network is Online."
only appears after the interface got configured via DHCP.

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: netplan.io 1.0-2ubuntu1
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
ApportVersion: 2.28.1-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: pass
Date: Sat Apr 27 23:50:37 2024
InstallationDate: Installed on 2024-04-26 (1 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release amd64 (20240423)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
SourcePackage: netplan.io
UpgradeStatus: No upgrade log present (probably fresh install)

[Lucas Nussbaum (lucas)]

It worked fine in Ubuntu 23.10: network-online.target is only reached after eno1 is configured.

# journalctl -b |grep -e network -e grid5000.mount
Apr 28 00:23:06 localhost kernel: drop_monitor: Initializing network drop monitor service
Apr 28 00:23:06 localhost systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
Apr 28 00:23:08 localhost systemd[1]: Starting cloud-init-local.service - Initial cloud-init job (pre-networking)...
Apr 28 00:23:09 localhost systemd[1]: Finished cloud-init-local.service - Initial cloud-init job (pre-networking).
Apr 28 00:23:09 localhost systemd[1]: Reached target network-pre.target - Preparation for Network.
Apr 28 00:23:09 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Apr 28 00:23:09 localhost systemd-networkd[1157]: lo: Link UP
Apr 28 00:23:09 localhost systemd-networkd[1157]: lo: Gained carrier
Apr 28 00:23:09 localhost systemd-networkd[1157]: Enumeration completed
Apr 28 00:23:09 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Configuring with /run/systemd/network/10-netplan-eno1.network.
Apr 28 00:23:09 localhost systemd[1]: Reached target network.target - Network.
Apr 28 00:23:09 localhost systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Link UP
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Gained carrier
Apr 28 00:23:10 localhost systemd-networkd[1157]: eno1: DHCPv4 address 172.16.66.58/20, gateway 172.16.79.254 acquired from 172.16.79.107
Apr 28 00:23:11 localhost systemd-networkd[1157]: eno1: Gained IPv6LL
Apr 28 00:23:11 localhost systemd[1]: Finished systemd-networkd-wait-online.service - Wait for Network to be Configured.
Apr 28 00:23:12 localhost systemd[1]: Reached target network-online.target - Network is Online.
Apr 28 00:23:13 localhost systemd[1]: Mounting grid5000.mount - /grid5000...
Apr 28 00:23:13 localhost dbus-daemon[1182]: [system] Activating systemd to hand-off: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.0' (uid=101 pid=1157 comm="/lib/systemd/systemd-networkd" label="unconfined")
Apr 28 00:23:13 localhost systemd[1]: Starting networkd-dispatcher.service - Dispatcher daemon for systemd-networkd...
Apr 28 00:23:13 localhost networkd-dispatcher[1197]: No valid path found for iwconfig
Apr 28 00:23:13 localhost networkd-dispatcher[1197]: No valid path found for iw
Apr 28 00:23:13 localhost systemd[1]: Started networkd-dispatcher.service - Dispatcher daemon for systemd-networkd.
Apr 28 00:23:13 localhost systemd-networkd[1157]: eno1: DHCPv6 address 2001:660:4406:500:3::3a/128 (valid for 23h 59min 59s, preferred for 14h 59min 59s)
Apr 28 00:23:14 gros-58 systemd[1]: Reloading requested from client PID 1345 (unit grid5000.mount)...
Apr 28 00:23:14 gros-58 systemd[1]: Mounted grid5000.mount - /grid5000.
Apr 28 00:23:22 gros-58 systemd[1541]: Listening on dirmngr.socket - GnuPG network certificate management daemon.

[Lucas Nussbaum (lucas)]

I think I understood this.

I have a udev rule to force the NIC name to eno1, which is not the driver's default (it is eno1np0). I did not regenerate the initramfs, so that rule did not propagate there.

What happens is:
- initramfs is executed. The NIC is renamed to eno1np0 (driver's default).
- systemd starts
- it runs the netplan generator
- the netplan generator sees its configuration for eno1, but does not match it to eno1np0.
- as a consequence, it does not generate the symlink (/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service)
- udev kicks in, renames the NIC to eno1, but it's too late.

That behaviour is caused by debian/patches/lp2060311/0010-wait-online-wait-for-existing-interfaces-only-and-do.patch. Removing that patch returns to the previous behaviour.

There are two potential workarounds:
- regenerate the initramfs, so that it knows about the new name for the interface and names it correctly before systemd starts.
- help netplan match the real NIC with its configuration, for example specifying the MAC address.
```
---
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: true
      match:
        macaddress: 98:03:9b:b0:a7:76
```

[Danilo Egea Gondolfo (danilogondolfo)]

Hi Lucas,

In your scenario the best approach would probably be relying on Netplan to rename the interface for you instead of creating an external udev rule.

I'd suggest you use "set-name: eno1" and continue to use "match" in your configuration. Something like this:

```
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1np0:
      dhcp4: true
      match:
        macaddress: 98:03:9b:b0:a7:76
      set-name: eno1
```

[Nick Grealy (nickgrealy)]

Hi all,

I didn't see a response to the workaround, so just dropping a message.

I had the same issue. Confirming that workaround - configuring my `/etc/netplan/50-cloud-init.yaml` to match the network interface based on macaddress (`ip a`), and applying the change (`netplan generate && netplan --debug apply`) solved the issue.

`netplan status` now shows the network interface has an IP address assigned by DHCP.

Thank you