network-online.target is reached before netplan-configured interfaces are online

Bug #2063973 reported by Lucas Nussbaum
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
netplan.io (Ubuntu)
Triaged
High
Unassigned

Bug Description

Hi,

Running Ubuntu 24.04 with netplan.io/netplan-generator 1.0-2ubuntu1.

I use a systemd unit to mount an NFS filesystem. That unit should only be activated when the network is ready. However, it looks like network-online.target is reached before netplan-configured interfaces are up, so the unit triggers too early and sometimes fail (because it cannot resolve the name of the remote server).

I configured netplan manually with:
```
# cat /etc/netplan/01-netcfg.yaml
---
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: true
```

I see the following during boot:

```
Apr 27 23:31:28 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Apr 27 23:31:28 localhost systemd-networkd[1190]: lo: Link UP
Apr 27 23:31:28 localhost systemd-networkd[1190]: lo: Gained carrier
Apr 27 23:31:28 localhost systemd-networkd[1190]: Enumeration completed
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Configuring with /run/systemd/network/10-netplan-eno1.network.
Apr 27 23:31:28 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Apr 27 23:31:28 localhost systemd[1]: Reached target network.target - Network.
Apr 27 23:31:28 localhost systemd[1]: systemd-networkd-wait-online.service - Wait for Network to be Configured was skipped because of an unmet condition check (ConditionPathIsSymbolicLink=/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service).
Apr 27 23:31:28 localhost systemd[1]: Reached target network-online.target - Network is Online.
Apr 27 23:31:28 localhost systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
Apr 27 23:31:28 localhost systemd[1]: Mounting grid5000.mount - /grid5000...
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Link UP
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: Gained carrier
Apr 27 23:31:28 localhost systemd-networkd[1190]: eno1: DHCPv4 address 172.16.66.53/20, gateway 172.16.79.254 acquired from 172.16.79.107
Apr 27 23:31:28 localhost systemd[1]: Starting networkd-dispatcher.service - Dispatcher daemon for systemd-networkd...
Apr 27 23:31:29 gros-53 systemd[1]: Starting rpc-statd.service - NFS status monitor for NFSv2/3 locking....
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Version 2.6.4 starting
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Flags: TI-RPC
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Failed to read /var/lib/nfs/state: Success
Apr 27 23:31:29 gros-53 rpc.statd[1439]: Initializing NSM state
Apr 27 23:31:29 gros-53 systemd[1]: Started rpc-statd.service - NFS status monitor for NFSv2/3 locking..
Apr 27 23:31:29 gros-53 mount[1441]: Created symlink /run/systemd/system/remote-fs.target.wants/rpc-statd.service → /usr/lib/systemd/system/rpc-statd.service.
Apr 27 23:31:29 gros-53 systemd[1]: Reloading requested from client PID 1441 ('systemctl') (unit grid5000.mount)...
Apr 27 23:31:29 gros-53 systemd[1]: Reloading...
Apr 27 23:31:30 gros-53 systemd[1]: Reloading finished in 194 ms.
Apr 27 23:31:30 gros-53 systemd[1]: Mounted grid5000.mount - /grid5000.
Apr 27 23:31:30 gros-53 nfsrahead[1497]: setting /grid5000 readahead to 128

The key message here seems to be:
Apr 27 23:31:28 localhost systemd[1]: systemd-networkd-wait-online.service - Wait for Network to be Configured was skipped because of an unmet condition check (ConditionPathIsSymbolicLink=/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service).
Why doesn't netplan generate that symlink? It seems it does generate it, but only later, when systemd reloads at 23:31:29 (according to 'stat' on the symlink).

It worked fine with Ubuntu 22.04:
"systemd[1]: Reached target Network is Online."
only appears after the interface got configured via DHCP.

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: netplan.io 1.0-2ubuntu1
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
ApportVersion: 2.28.1-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: pass
Date: Sat Apr 27 23:50:37 2024
InstallationDate: Installed on 2024-04-26 (1 days ago)
InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release amd64 (20240423)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
SourcePackage: netplan.io
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Lucas Nussbaum (lucas) wrote :
Revision history for this message
Lucas Nussbaum (lucas) wrote :

It worked fine in Ubuntu 23.10: network-online.target is only reached after eno1 is configured.

# journalctl -b |grep -e network -e grid5000.mount
Apr 28 00:23:06 localhost kernel: drop_monitor: Initializing network drop monitor service
Apr 28 00:23:06 localhost systemd[1]: Listening on systemd-networkd.socket - Network Service Netlink Socket.
Apr 28 00:23:08 localhost systemd[1]: Starting cloud-init-local.service - Initial cloud-init job (pre-networking)...
Apr 28 00:23:09 localhost systemd[1]: Finished cloud-init-local.service - Initial cloud-init job (pre-networking).
Apr 28 00:23:09 localhost systemd[1]: Reached target network-pre.target - Preparation for Network.
Apr 28 00:23:09 localhost systemd[1]: Starting systemd-networkd.service - Network Configuration...
Apr 28 00:23:09 localhost systemd-networkd[1157]: lo: Link UP
Apr 28 00:23:09 localhost systemd-networkd[1157]: lo: Gained carrier
Apr 28 00:23:09 localhost systemd-networkd[1157]: Enumeration completed
Apr 28 00:23:09 localhost systemd[1]: Started systemd-networkd.service - Network Configuration.
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Configuring with /run/systemd/network/10-netplan-eno1.network.
Apr 28 00:23:09 localhost systemd[1]: Reached target network.target - Network.
Apr 28 00:23:09 localhost systemd[1]: Starting systemd-networkd-wait-online.service - Wait for Network to be Configured...
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Link UP
Apr 28 00:23:09 localhost systemd-networkd[1157]: eno1: Gained carrier
Apr 28 00:23:10 localhost systemd-networkd[1157]: eno1: DHCPv4 address 172.16.66.58/20, gateway 172.16.79.254 acquired from 172.16.79.107
Apr 28 00:23:11 localhost systemd-networkd[1157]: eno1: Gained IPv6LL
Apr 28 00:23:11 localhost systemd[1]: Finished systemd-networkd-wait-online.service - Wait for Network to be Configured.
Apr 28 00:23:12 localhost systemd[1]: Reached target network-online.target - Network is Online.
Apr 28 00:23:13 localhost systemd[1]: Mounting grid5000.mount - /grid5000...
Apr 28 00:23:13 localhost dbus-daemon[1182]: [system] Activating systemd to hand-off: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.0' (uid=101 pid=1157 comm="/lib/systemd/systemd-networkd" label="unconfined")
Apr 28 00:23:13 localhost systemd[1]: Starting networkd-dispatcher.service - Dispatcher daemon for systemd-networkd...
Apr 28 00:23:13 localhost networkd-dispatcher[1197]: No valid path found for iwconfig
Apr 28 00:23:13 localhost networkd-dispatcher[1197]: No valid path found for iw
Apr 28 00:23:13 localhost systemd[1]: Started networkd-dispatcher.service - Dispatcher daemon for systemd-networkd.
Apr 28 00:23:13 localhost systemd-networkd[1157]: eno1: DHCPv6 address 2001:660:4406:500:3::3a/128 (valid for 23h 59min 59s, preferred for 14h 59min 59s)
Apr 28 00:23:14 gros-58 systemd[1]: Reloading requested from client PID 1345 (unit grid5000.mount)...
Apr 28 00:23:14 gros-58 systemd[1]: Mounted grid5000.mount - /grid5000.
Apr 28 00:23:22 gros-58 systemd[1541]: Listening on dirmngr.socket - GnuPG network certificate management daemon.

Revision history for this message
Lucas Nussbaum (lucas) wrote :

I think I understood this.

I have a udev rule to force the NIC name to eno1, which is not the driver's default (it is eno1np0). I did not regenerate the initramfs, so that rule did not propagate there.

What happens is:
- initramfs is executed. The NIC is renamed to eno1np0 (driver's default).
- systemd starts
- it runs the netplan generator
- the netplan generator sees its configuration for eno1, but does not match it to eno1np0.
- as a consequence, it does not generate the symlink (/run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service)
- udev kicks in, renames the NIC to eno1, but it's too late.

That behaviour is caused by debian/patches/lp2060311/0010-wait-online-wait-for-existing-interfaces-only-and-do.patch. Removing that patch returns to the previous behaviour.

There are two potential workarounds:
- regenerate the initramfs, so that it knows about the new name for the interface and names it correctly before systemd starts.
- help netplan match the real NIC with its configuration, for example specifying the MAC address.
```
---
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: true
      match:
        macaddress: 98:03:9b:b0:a7:76
```

Lukas Märdian (slyon)
tags: added: foundations-todo network-online-ordering
Changed in netplan.io (Ubuntu):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Danilo Egea Gondolfo (danilogondolfo) wrote :

Hi Lucas,

In your scenario the best approach would probably be relying on Netplan to rename the interface for you instead of creating an external udev rule.

I'd suggest you use "set-name: eno1" and continue to use "match" in your configuration. Something like this:

```
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1np0:
      dhcp4: true
      match:
        macaddress: 98:03:9b:b0:a7:76
      set-name: eno1
```

Revision history for this message
Nick Grealy (nickgrealy) wrote :

Hi all,

I didn't see a response to the workaround, so just dropping a message.

I had the same issue. Confirming that workaround - configuring my `/etc/netplan/50-cloud-init.yaml` to match the network interface based on macaddress (`ip a`), and applying the change (`netplan generate && netplan --debug apply`) solved the issue.

`netplan status` now shows the network interface has an IP address assigned by DHCP.

Thank you

Revision history for this message
Lukas Märdian (slyon) wrote :

More improvements landing via https://github.com/canonical/netplan/pull/482

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.