systemd wait-online now times out after jammy and lunar upgrade

[Jamie Strandboge]

[NOTE]

If you are running a desktop system and you see this issue, you should run:

$ systemctl disable --now systemd-networkd.service

This will disable systemd-networkd and associated units, including systemd-networkd-wait-online.service. NetworkManager and systemd-networkd should not be running at the same time. On desktop, NetworkManager is the default network stack.

[Impact]

When all interfaces are "not required for online", e.g. when they are marked "optional: true" in netplan, systemd-networkd-wait-online will timeout. Or, in other words, systemd-networkd-wait-online will timeout even though all interfaces are ignored, hence none of them will ever be marked as "ready." Depending on what units depend on network-online.target, this can delay boot by 120 seconds (the default timeout for systemd-networkd-wait-online).

[Test Plan]

1. Create a new LXD container. These instructions assume jammy is the release, but the same can be done for lunar.

$ lxc launch ubuntu-daily:jammy jammy
$ lxc exec jammy bash

2. Once in the container, modify the default /etc/netplan/10-lxc.yaml so that eth0 is configured with "optional: true":

$ vi /etc/netplan/50-cloud-init.yaml # Use whatever editor you like
$ cat /etc/netplan/50-cloud-init.yaml
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp-identifier: mac
      optional: true

3. Re-generate and apply the netplan configuration.

$ netplan generate
$ netplan apply

4. Manually run systemd-networkd-wait-online, and observe that all links are ignored, and the command times out:

$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link lo(1)
Found link eth0(19)
lo: link is ignored
eth0: link is ignored
Timeout occurred while waiting for network connectivity.

[Where problems could occur]

This patch partially re-instates a patch remove in bug 1982218. However, instead of exiting if all links are unmanaged, we exit if all links are ignored in manager_configured(). If the patch was wrong, we may re-introduce bug 1982218, so as part of this SRU verification, that bug should be tested too. Any other regressions would also be related to systemd-networkd-wait-online behavior.

[Original Description]

On Ubuntu 22.04 desktop system using network-manager and upgrading to systemd 249.11-0ubuntu3.10, wait-online now times out which prevents logins (GDM, ssh, console) until it does time out. This seems to be introduced by the change for https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218.

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1982218/comments/21 also mentioned the problem on Lunar.

[Steve Langasek (vorlon)]

> On Ubuntu 22.04 desktop system using network-manager

To be clear, does this mean you have no network interfaces that are configured to use networkd?

[Nick Rosbrook (enr0n)]

If you are using NetworkManager, then systemd-networkd.service (and associated units like systemd-networkd-wait-online.service) should NOT be enabled. With the caveat that I am not sure why you have systemd-networkd enabled in the first place, I would recommend that you simply disable it:

$ systemctl disable --now systemd-networkd.service

This will also have the effect of disabling systemd-networkd-wait-online.service.

[Dave Jones (waveform)]

Also appears to be affecting the Pi server images (which have no network manager and only use networkd for networking); I've only checked lunar so far.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Biech (tikurion-n)]

I'm also affected by this: I'm on Ubuntu 22.04.3 for Raspberry Pi and after updating to 249.11-0ubuntu3.10 on boot I get:

----

Sep 18 19:16:33 pi4-ubuntu systemd[1]: Starting Wait for Network to be Configured...
Sep 18 19:19:29 pi4-ubuntu systemd-networkd-wait-online[1319]: Timeout occurred while waiting for network connectivity.
Sep 18 19:19:29 pi4-ubuntu systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 19:19:29 pi4-ubuntu systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Sep 18 19:19:29 pi4-ubuntu systemd[1]: Failed to start Wait for Network to be Configured.

----

Output of networkctl:

----

IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured

2 links listed.

----

Running `SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=5 --any` yields:

----

Found link 2
Found link 1
eth0: link is ignored
lo: link is ignored
Timeout occurred while waiting for network connectivity.

----

Which makes no sense, given eth0 is up and running.

I've made no changes to the default network settings and everything worked up until the update. This also doesn't bother me TOO much, since after the timeout of about 2 minutes, things continue as usual. Just a bit annoying and I was honestly wondering whether I was going crazy before I (luckily) stumbled upon this.

[Jamie Strandboge (jdstrand)]

>> On Ubuntu 22.04 desktop system using network-manager

> To be clear, does this mean you have no network interfaces that are configured to use networkd?

Hey Steve :)

So, this system is quite old. I think the first install was 16.04 and it went through a bunch of upgrades (mostly interim until 20.04 and then to 22.04). It would not surprise me if things weren't pristine. I'll give some info and you can tell me what else to provide:

$ cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager

$ sudo netplan get all
network:
  version: 2
  renderer: NetworkManager

$ ls /etc/systemd/network
$

$ cat /etc/systemd/networkd.conf
# This file is part of systemd.
...
# See networkd.conf(5) for details.

[Network]
#SpeedMeter=no
#SpeedMeterIntervalSec=10sec
#ManageForeignRoutingPolicyRules=yes
#ManageForeignRoutes=yes
#RouteTable=

[DHCPv4]
#DUIDType=vendor
#DUIDRawData=

[DHCPv6]
#DUIDType=vendor
#DUIDRawData=

$ sudo systemctl status systemd-networkd
● systemd-networkd.service - Network Configuration
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-09-17 12:13:48 CDT; 1 day 2h ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
   Main PID: 1442 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 18857)
     Memory: 2.7M
        CPU: 1.387s
     CGroup: /system.slice/systemd-networkd.service
             └─1442 /lib/systemd/systemd-networkd

Sep 17 14:33:42 <host> systemd-networkd[1442]: wlp58s0: Lost carrier
Sep 17 14:33:43 <host> systemd-networkd[1442]: wlp58s0: Gained carrier
Sep 17 16:44:57 <host> systemd-networkd[1442]: lxd0: Link UP
Sep 17 16:45:02 <host> systemd-networkd[1442]: veth8afa41ff: Link UP
Sep 17 16:45:02 <host> systemd-networkd[1442]: veth8afa41ff: Gained carrier
Sep 17 16:45:02 <host> systemd-networkd[1442]: lxd0: Gained carrier
Sep 17 16:45:02 <host> systemd-networkd[1442]: lxd0: Gained IPv6LL
Sep 17 17:32:49 <host> systemd-networkd[1442]: wlp58s0: Lost carrier
Sep 17 17:32:50 <host> systemd-networkd[1442]: wlp58s0: Connected WiFi access point: <ap> (<mac>)
Sep 17 17:32:50 <host> systemd-networkd[1442]: wlp58s0: Gained carrier

I don't recall enabling systemd-networkd, but a lot has happened over the last 7 years, 4.5 of those while developing Ubuntu. :)

[Jamie Strandboge (jdstrand)]

> If you are using NetworkManager, then systemd-networkd.service (and associated units like systemd-networkd-wait-online.service) should NOT be enabled. With the caveat that I am not sure why you have systemd-networkd enabled in the first place, I would recommend that you simply disable it:

> $ systemctl disable --now systemd-networkd.service

> This will also have the effect of disabling systemd-networkd-wait-online.service.

Ok, I've done this. It is easy enough to re-enable if I need to test anything wrt the bug. This resolved the issue for me.

[pernacentus (pernacentus)]

It also affects Ubuntu Server 22.04 after updating to 249.11-0ubuntu3.10, since I'm on a headless server, I'm not using NetworkManager.
On boot systemd-networkd-wait-online hangs and fails after a timeout.

Here's my netplan configuration:

----

$ cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp2s0:
      addresses: [192.168.178.116/24]
      nameservers:
        addresses: [1.1.1.1,1.0.0.1]
      routes:
        - to: default
          via: 192.168.178.1
      dhcp4: false
      dhcp6: false
      optional: true
    enp3s0:
      dhcp4: false
      dhcp6: false
      optional: true
    enp4s0:
      dhcp4: false
      dhcp6: false
      optional: true
    enp5s0:
      dhcp4: false
      dhcp6: false
      optional: true
    enp6s0:
      dhcp4: false
      dhcp6: false
      optional: true
    enp7s0:
      dhcp4: false
      dhcp6: false
      optional: true
  version: 2

----

cat /run/systemd/network/10-netplan-enp2s0.network

----

[Match]
Name=enp2s0

[Link]
RequiredForOnline=no

[Network]
LinkLocalAddressing=ipv6
Address=192.168.178.116/24
DNS=1.1.1.1
DNS=1.0.0.1

[Route]
Destination=0.0.0.0/0
Gateway=192.168.178.1

---

Other interfaces have also RequiredForOnline=no, hence they should be ignored by systemd-networkd-wait-online and it should not hang at boot.

Also for me running SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=5 --any yelds:

---

Found link 7
Found link 6
Found link 5
Found link 4
Found link 3
Found link 2
Found link 1
enp4s0: link is ignored
enp6s0: link is ignored
enp5s0: link is ignored
enp2s0: link is ignored
enp3s0: link is ignored
lo: link is ignored
enp7s0: link is ignored

---

[Nick Rosbrook (enr0n)]

I will work on re-instating the previous patch in some fashion to fix this bug without regressing 1982218.

[Nick Rosbrook (enr0n)]

This behavior is not present in Mantic, so nothing to do there. I have staged fixes for Jammy and Lunar.

[Chris Halse Rogers (raof)]

Hello Jamie, or anyone else affected,

Accepted systemd into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/252.5-2ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Jamie, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (252.5-2ubuntu3.2) for lunar have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.6.0ubuntu0.1 (armhf)
dbus/1.14.4-1ubuntu1 (i386)
fwupd/1.8.12-2 (ppc64el)
indicator-session/17.3.20+21.10.20210613.1-0ubuntu1 (armhf)
libsoup2.4/2.74.3-1 (amd64)
libsoup3/3.4.0-1 (amd64)
linux-lowlatency/6.2.0-1014.14 (arm64)
mariadb/1:10.11.2-1 (armhf)
mariadb-10.6/1:10.6.11-1 (armhf)
nut/2.8.0-7 (amd64, armhf)
ubuntu-drivers-common/1:0.9.7.1.1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/lunar/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Michael Biech (tikurion-n)]

Thank you everyone for your speedy work! I have verified "249.11-0ubuntu3.11" fixes the issue on Jammy for me! Very much appreciated.

Steps taken:

----

# 1)
cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

# 2)
apt update

# 3)
apt-get install --only-upgrade systemd

# 4)
reboot

# At this point, the system boots up fine, without waiting for the timeout as before

# Just to verify – 5)
SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=5 --any

# Output:
Found link 2
Found link 1
lo: link is ignored
eth0: link is ignored

# Returns immediately with an exit code of "0" :)

----

Awesome! Again: thank you very much!!

[Ron Economos (w6rz)]

Tested 249.11-0ubuntu3.11 on Ubuntu 22.04 (jammy) for riscv64 (HiFive Unmatched). The system now boots normally.

$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
rtnl: message parse - overwriting repeated attribute
Found link 2
Found link 1
eth0: link is ignored
lo: link is ignored

LGTM.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.11) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

casync/2+20201210-1build1 (arm64, ppc64el)
dbus/1.12.20-2ubuntu4.1 (armhf)
libsoup3/3.0.7-0ubuntu1 (armhf)
linux-aws-6.2/6.2.0-1013.13~22.04.1 (arm64)
linux-lowlatency-hwe-5.19/5.19.0-1030.30 (arm64)
linux-lowlatency-hwe-6.2/6.2.0-1014.14~22.04.1 (arm64)
linux-nvidia-6.2/6.2.0-1010.10 (amd64)
linux-nvidia-tegra/5.15.0-1017.17 (arm64)
linux-nvidia-tegra-igx/5.15.0-1004.4 (arm64)
linux-xilinx-zynqmp/5.15.0-1023.27 (arm64)
mutter/42.9-0ubuntu4 (arm64)
netplan.io/0.106.1-7ubuntu0.22.04.2 (amd64, arm64, ppc64el, s390x)
stunnel4/3:5.63-1build1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Robert Hrovat (robi-hipnos)]

Tested 249.11-0ubuntu3.11 on Ubuntu 22.04 (jammy) and system still wait for network to be configured

$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 3
Found link 2
Found link 1
enp2s0: link is not managed by networkd (yet?).
wlp3s0: link is not managed by networkd (yet?).
lo: link is ignored
enp2s0: link is not managed by networkd (yet?).
wlp3s0: link is not managed by networkd (yet?).
lo: link is ignored
Timeout occurred while waiting for network connectivity.

[Nick Rosbrook (enr0n)]

Robert - are you on on desktop or server? If the links are not managed by systemd-networkd, it seems like you would be running NetworkManager? What does this show for you?

$ systemctl status systemd-networkd
$ systemctl status NetworkManager

[Tais P. Hansen (taisph)]

Tested 252.5-2ubuntu3.2 on Ubuntu 23.04 (lunar) and it still hangs for 2 minutes during boot.

$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 7
Found link 6
Found link 5
Found link 4
Found link 3
Found link 2
Found link 1
lo: link is ignored
wlp6s0: link is not managed by networkd (yet?).
virbr2: link is not managed by networkd (yet?).
enp5s0: link is not managed by networkd (yet?).
enp4s0: link is not managed by networkd (yet?).
virbr0: link is not managed by networkd (yet?).
tailscale0: link is not managed by networkd (yet?).
Timeout occurred while waiting for network connectivity.

Desktop system.

Both systemd-networkd and NetworkManager services are running and active.

[Nick Rosbrook (enr0n)]

Tais - you should not have both services running. For you the solution should be:

$ systemctl disable --now systemd-networkd.service

This will also disable associated units, including systemd-networkd-wait-online.service.

[Nick Rosbrook (enr0n)]

I have verified the fix using systemd 252.5-2ubuntu3.2 in lunar-proposed:

Creating a fresh container:

nr@six:~$ lxc launch ubuntu-daily:lunar lunar
Creating lunar
Starting lunar
nr@six:~$ lxc exec lunar bash

In the container, after enabling -proposed:

root@lunar:~# apt update && apt install -y systemd
[...]
root@lunar:~# apt-cache policy systemd
systemd:
  Installed: 252.5-2ubuntu3.2
  Candidate: 252.5-2ubuntu3.2
  Version table:
 *** 252.5-2ubuntu3.2 500
        500 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     252.5-2ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages
     252.5-2ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root@lunar:~# vi /etc/netplan/50-cloud-init.yaml
root@lunar:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true
            optional: true
root@lunar:~# netplan generate
root@lunar:~# netplan apply
Cannot call openvswitch: ovsdb-server.service is not running.
root@lunar:~# time SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-networkd-wait-online --timeout=5
Found link 13
Found link 1
eth0: link is ignored
lo: link is ignored

real 0m0.003s
user 0m0.003s
sys 0m0.000s
root@lunar:~# echo $?
0

I also ran the test from bug 1982218 again to ensure that does not regress:

On the host, I have clean-lunar-amd64, and detached it's default network interface:

nr@six:~$ virsh detach-interface clean-lunar-amd64 network
Interface detached successfully

Then, in the VM, configure systemd-networkd to use DHCP on all en* interfaces:

root@clean-lunar-amd64:/home/nr# apt-cache policy systemd
systemd:
  Installed: 252.5-2ubuntu3.2
  Candidate: 252.5-2ubuntu3.2
  Version table:
 *** 252.5-2ubuntu3.2 500
        500 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     252.5-2ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu lunar-updates/main amd64 Packages
     252.5-2ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root@clean-lunar-amd64:/home/nr# cat > /etc/systemd/network/10-dhcp.network << EOF
[Match]
Name=en*

[Network]
DHCP=yes
EOF
root@clean-lunar-amd64:/home/nr# systemctl restart systemd-networkd

Now, back in the host, I attach a network interface without DHCP:

nr@six:~$ virsh attach-interface clean-lunar-amd64 network no-dhcp
Interface attached successfully

In the VM, check that wait-online times out since the interface does not get configured:

root@clean-lunar-amd64:/home/nr# /lib/systemd/systemd-networkd-wait-online --any --timeout=5
Timeout occurred while waiting for network connectivity.
root@clean-lunar-amd64:/home/nr# networkctl
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loo...

[Nick Rosbrook (enr0n)]

I have verified the fix using systemd 249.11-0ubuntu3.11 from jammy-proposed:

Creating a fresh container:

nr@six:~$ lxc launch ubuntu-daily:jammy jammy
Creating jammy
Starting jammy
nr@six:~$ lxc exec jammy bash

Within the container, after enabling -proposed:

root@jammy:~# vi /etc/apt/sources.list
root@jammy:~# apt update && apt install systemd -y
[ ... ]
root@jammy:~# apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.11
  Candidate: 249.11-0ubuntu3.11
  Version table:
 *** 249.11-0ubuntu3.11 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@jammy:~# vi /etc/netplan/50-cloud-init.yaml
root@jammy:~# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true
            optional: true
root@jammy:~# netplan generate
root@jammy:~# netplan apply
root@jammy:~# time SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=5
Found link 11
Found link 1
eth0: link is ignored
lo: link is ignored

real 0m0.004s
user 0m0.003s
sys 0m0.000s
root@jammy:~# echo $?
0

I also ran the test from bug 1982218 again to ensure that does not regress:

On the host, I have clean-jammy-amd64, and detached it's default network interface:

nr@six:~$ virsh detach-interface clean-jammy-amd64 network
Interface detached successfully

Then, in the VM, configure systemd-networkd to use DHCP on all en* interfaces:

nr@clean-jammy-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.11
  Candidate: 249.11-0ubuntu3.11
  Version table:
 *** 249.11-0ubuntu3.11 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
nr@clean-jammy-amd64:~$ sudo su
root@clean-jammy-amd64:/home/nr# cat > /etc/systemd/network/10-dhcp.network << EOF
[Match]
Name=en*

[Network]
DHCP=yes
EOF
root@clean-jammy-amd64:/home/nr# systemctl restart systemd-networkd

Now, back in the host, I attach a network interface without DHCP:

nr@six:~$ virsh attach-interface clean-jammy-amd64 network no-dhcp
Interface attached successfully

In the VM, check that wait-online times out since the interface does not get configured:

root@clean...

[Tais P. Hansen (taisph)]

Thanks for the tip. Not sure why both were enabled but disabling systemd-networkd.service fixes the long wait during boot for me.

[Birgit Edel (biredel)]

re: "you should not have both services running"

Is this documented somewhere? What software or script, during system installation, is meant to ensure this? I have not seen warnings emitted from postinst (as added for /etc/network/interfaces) or service start.

I am not certain it needs to be, as
A) wait for any/all managed interfaces in configuring state and
B) wait for any/all unmanaged interfaces
appear to be sufficiently distinguishable cases to let systemd do A without also doing B, with or without the (orthogonal) "any" modifier.

[Steve Langasek (vorlon)]

It is not a typical configuration to have both services running.

It makes sense to have both services running, IF you have configured separate network interfaces to be managed by each. But this is a very uncommon configuration.

The expectation is that the services are dynamically enabled only if netplan has configuration for the corresponding backend; or if the user has chosen to manually enable it bypassing netplan.

[pernacentus (pernacentus)]

Tested 249.11-0ubuntu3.11 on Ubuntu Server 22.04 (Jammy) and now I have no delay on boot.
Also the following command now returns immediately:
$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 9
Found link 8
Found link 7
Found link 6
Found link 5
Found link 4
Found link 3
Found link 2
Found link 1
enp4s0: link is ignored
eno1: link is ignored
enp3s0: link is ignored
enp5s0: link is ignored
lo: link is ignored
eno2: link is ignored
enp7s0: link is ignored
enp2s0: link is ignored
enp6s0: link is ignored

Thanks!

[Mitchell J. O'Sullivan (mosullivan93)]

Hi folks,

I landed here while investigating issues running Ubuntu 22.04 (Jammy) on WSL2. I tested the proposed version and have mixed results.

NetworkManager is not installed. systemd-network is enabled, but I don't know if WSL actually uses it.

Like 249.11-0ubuntu3.10, 249.11-0ubuntu3.11 also times out for me when run as follows:
mosullivan@MITCH-PC:~$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 2
Found link 1
lo: link is ignored
eth0: link is not managed by networkd (yet?).
Timeout occurred while waiting for network connectivity.

However, on 249.11-0ubuntu3.11, but not on 249.11-0ubuntu3.10, I can use the --ignore option and the program exits immediately and successfully when appropriate:
mosullivan@MITCH-PC:~$ SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --ignore=eth0 --timeout=10
Found link 2
Found link 1
lo: link is ignored
eth0: link is ignored

This is probably indicative that Microsoft ought to tweak their integration but I thought I'd share anyway. I'm fine for now to override ExecStart (i.e. to include --ignore) so that things work smoothly again using the latest version.

Kind regards,
Mitchell O'Sullivan

[Nick Rosbrook (enr0n)]

I believe the remaining autopkgtest failures are unrelated.

For lunar[1], we have ubuntu-drivers-common failing on ppc64el[2]. We can see the same failure on a different test run without a systemd trigger[3].

For jammy[4], we have linux-nvidia[5] and linux-nvidia-tegra-igx[6], each failing their "rebuild" test. This happens frequently and is unrelated to the systemd change. Finally, there is netplan.io failing its "tunnels" test[7]. This is failing without systemd triggers too[8], and I checked with @danilogondolfo to confirm that this is a separate issue.

[1] https://ubuntu-archive-team.ubuntu.com/proposed-migration/lunar/update_excuses.html#systemd
[2] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/u/ubuntu-drivers-common/20230925_224541_eee1a@/log.gz
[3] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/u/ubuntu-drivers-common/20230815_053853_379e2@/log.gz
[4] https://ubuntu-archive-team.ubuntu.com/proposed-migration/jammy/update_excuses.html#systemd
[5] https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/l/linux-nvidia-6.2/20230926_000856_1a567@/log.gz
[6] https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/l/linux-nvidia-tegra-igx/20230926_010738_40534@/log.gz
[7] https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/n/netplan.io/20230925_232048_829bd@/log.gz
[8] https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/n/netplan.io/20230907_195833_5846d@/log.gz

[Dave Jones (waveform)]

Seeing the same results as comment 27 with the Pi lunar (23.04) image. After installing -proposed and checking systemd-networkd-wait-online, with wlan0 undefined in the netplan configuration (but eth0 set as the default DHCP and optional), it still waits for the full 2 minutes, presumably because wlan0 is not explicitly marked as "not required" as it's not a defined interface in the netplan config?

[mev (mev1)]

Dave - This happens to me on Pi as well. I have wlan0 fully disabled, so that doesn't appear to be the issue.

[Nick Rosbrook (enr0n)]

Dave - yeah that's the expect outcome with this patch.

[Werner Harnisch (wharnisch2)]

Did the update via jammy-proposed. Sorry, but 249.11-0ubuntu3.11 does NOT fix the issue of "A start job is running for Wait For Network to be Configured hanging at boot for 2 minutes in Ubuntu 22.04 servers" for me on Ubuntu 22.04 server.

[Werner Harnisch (wharnisch2)]

Workaround is to remove "optional: true" from netplan settings, in this case the issue does not occur.

[Nick Rosbrook (enr0n)]

Werner - it sounds like you might have been having a different issue. Without the debug level output from systemd-networkd-wait-online, it's hard to tell what caused the timeout. Glad you have a workaround, though.

[Werner Harnisch (wharnisch2)]

@Nick

this is the syslogd output:

Oct 8 09:33:37 idefix systemd-networkd-wait-online[668]: Timeout occurred while waiting for network connectivity.
Oct 8 09:33:37 idefix systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Oct 8 09:33:37 idefix systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.

How can I get 'debug level output from systemd-networkd-wait-online'?

[Werner Harnisch (wharnisch2)]

@Nick

think I managed to get 'debug level output from systemd-networkd-wait-online' with a little help from a friend:

Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: Found link 4
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: Found link 3
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: Found link 2
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: Found link 1
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:42 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:43 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:43 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:43 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:43 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:45 idefix systemd-networkd-wait-online[676]: lo: link is ignored
Oct 8 10:00:45 idefix systemd-networkd-wait-online[676]: enp2s0: link is ignored
Oct 8 10:00:45 idefix systemd-networkd-wait-online[676]: wlo1: link is not managed by networkd (yet?).
Oct 8 10:00:45 idefix systemd-networkd-wait-online[676]: enp1s0: link is ignored
Oct 8 10:00:45 idefix syste...

[Shaun.M (shaun-as-online)]

I'm having the same effect like Werner. On a vm with jammy server, plain install - after an update today systemd-networkd delays booting. After removing the "optional: true" statement, the boot behaviour is normal.

Strange enough, thought this statement should do just the opposite.

[Juerg Haefliger (juergh)]

This introduces an unacceptable regression on Raspberry Pis. An interface that is *not* listed must not cause a 2 min timeout.

[Chris Halse Rogers (raof)]

(SRU team member): It's not entirely clear to me what the status of this bug is? It's marked as verification-done and all the autopkgtests are clear, but there's a lot of “it doesn't work” chatter. Let's see if I can summarise:

Werner seems to be seeing a hanging problem with 249.11-0ubuntu3.11. Looking at the debug information, it seems that there's a *non*-ignored interface present, and so they're not running into *this* bug, but are probably hitting the same problem reported on the Pi images?

Shaun (#37) doesn't specify the version, but I infer that they're using the version in -updates, not -proposed, so they're probably hitting the bug this upload is fixing.

So I *think* the next step here would be to release *this* package to -updates, as it seems to fix what it claims to fix without regressing anything else *over what's already in -updates*, and then file *another* bug (possibly against netplan?) to cover the “the real interface is marked as optional: true, but there is another, unmanaged interface” problem?

The assumption here is that this is *not* a regression over what's already in -updates - @Werner, @Dave, can you confirm that?

[BloodyIron (bloodyiron)]

I just upgraded a core system from Ubuntu 18.04->20.04->22.04, and this is somehow the only system that has this 2minute + wait bug.

How exactly was this not handled automatically in the upgrade process for 20.04->22.04? Isn't that the whole point of that automation?

I'm not entirely sure right now what I _should_ do to correct this... :/ this sucks

[Andreas Hasenack (ahasenack)]

Hi @enr0n, could you please comment on Chris's remark above about the state of this SRU?

[Oscar Guertler (oguertlertt)]

Currently experiencing the same issue as @wharnisch2 on ubuntu 22.04.3 5.15.0-86-generic with systemd 249 (249.11-0ubuntu3.10) on a somewhat fresh install (few weeks old)

[Isaac True (itrue)]

systemd 249.11-0ubuntu3.11 does *not* solve the issue for me. I seem to be running into the same situation as Werner in #32 where `optional: true` from the netplan config is not being respected.

systemd-networkd-wait-online seems to ignore `RequiredForOnline=no` in the .network config.

Netplan config:

ubuntu@ubuntu:~$ sudo netplan get
network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true
      dhcp6: true
    pfe:
      match:
        name: "pfe*"
      optional: true
      dhcp4: true
      dhcp6: true

systemd-networkd configs:

ubuntu@ubuntu:~$ grep '' /run/systemd/network/10-netplan-*
/run/systemd/network/10-netplan-eth0.network:[Match]
/run/systemd/network/10-netplan-eth0.network:Name=eth0
/run/systemd/network/10-netplan-eth0.network:
/run/systemd/network/10-netplan-eth0.network:[Link]
/run/systemd/network/10-netplan-eth0.network:RequiredForOnline=no
/run/systemd/network/10-netplan-eth0.network:
/run/systemd/network/10-netplan-eth0.network:[Network]
/run/systemd/network/10-netplan-eth0.network:DHCP=yes
/run/systemd/network/10-netplan-eth0.network:LinkLocalAddressing=ipv6
/run/systemd/network/10-netplan-eth0.network:
/run/systemd/network/10-netplan-eth0.network:[DHCP]
/run/systemd/network/10-netplan-eth0.network:RouteMetric=100
/run/systemd/network/10-netplan-eth0.network:UseMTU=true
/run/systemd/network/10-netplan-pfe.network:[Match]
/run/systemd/network/10-netplan-pfe.network:Name=pfe*
/run/systemd/network/10-netplan-pfe.network:
/run/systemd/network/10-netplan-pfe.network:[Link]
/run/systemd/network/10-netplan-pfe.network:RequiredForOnline=no
/run/systemd/network/10-netplan-pfe.network:
/run/systemd/network/10-netplan-pfe.network:[Network]
/run/systemd/network/10-netplan-pfe.network:DHCP=yes
/run/systemd/network/10-netplan-pfe.network:LinkLocalAddressing=ipv6
/run/systemd/network/10-netplan-pfe.network:
/run/systemd/network/10-netplan-pfe.network:[DHCP]
/run/systemd/network/10-netplan-pfe.network:RouteMetric=100
/run/systemd/network/10-netplan-pfe.network:UseMTU=true

eth0 is configured:
ubuntu@ubuntu:~$ networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 can0 can off unmanaged
  4 can1 can off unmanaged
  5 llcecan0 can off unmanaged
  6 llcecan1 can off unmanaged
  7 llcecan2 can off unmanaged
  8 llcecan3 can off unmanaged
  9 llcecan4 can off unmanaged
 10 llcecan5 can off unmanaged
 11 llcecan6 can off unmanaged
 12 llcecan7 can off unmanaged
 13 llcecan8 can off unmanaged
 14 llcecan9 can off unmanaged
 15 llcecan10 can off unmanaged
 16 llcecan11 can off unmanaged
 17 llcecan13 can off unmanaged
 18 llcecan14 can off unmanaged
 19 llcecan15 can off unmanaged
 20 pfe0 ether degraded configuring
 21 pfe1 ether no-carrier configuring
 22 pfe2 ether no-carrier configuring

22 links listed.

systemd-networkd-wait-online times out:

ubun...

[Nick Rosbrook (enr0n)]

Chris, Andreas, SRU team - In short, I think this should be released to -updates, yes. From looking through the various comments I have observed the same that either (a) a user is not testing with -proposed, or (b) they are in a slightly different situation (but see the timeout nonetheless, hence find this bug report).

[BloodyIron (bloodyiron)]

systemd "249.11-0ubuntu3.11" "fixed" it for me, thanks to those posting the solution.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.11

---------------
systemd (249.11-0ubuntu3.11) jammy; urgency=medium

  * wait-online: exit early if all links are ignored (LP: #2036358)
    File: debian/patches/lp2036358-wait-online-exit-early-if-all-links-are-ignored.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=fcf9af1efaa904b9e587a3bfa76814d2601d05f3

 -- Nick Rosbrook <email address hidden> Tue, 19 Sep 2023 12:57:00 -0400

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 252.5-2ubuntu3.2

---------------
systemd (252.5-2ubuntu3.2) lunar; urgency=medium

  * wait-online: exit early if all links are ignored (LP: #2036358)
    File: debian/patches/lp2036358-wait-online-exit-early-if-all-links-are-ignored.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b8b372b449c9b3cfbf556bbe5375da99b5735db

 -- Nick Rosbrook <email address hidden> Tue, 19 Sep 2023 12:50:07 -0400

[BloodyIron (bloodyiron)]

I'm now seeing a workable version of systemd available (but kept back?) on my 22.04 LTS systems without having to enasble proposed on them. This version (249.11-0ubuntu3.11) addresses the long boot problem, but I've had to tell apt directly to install systemd to trigger the update. For anyone who needs this right now, IMO go ahead and install it as I have not seen any side-effects. But for those who don't yet have the problem, updating will probably eventually apply the update to your system (assuming you update regularly of course).

Thanks devs!

[hu (peerlnk)]

systemd 249.11-0ubuntu3.11 bug still exists

root@ubuntu:~# netplan get
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      optional: true
      dhcp-identifier: "mac"

root@ubuntu:~# networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 eth1 ether off unmanaged

3 links listed.

root@ubuntu:~# systemctl status systemd-networkd-wait-online.service
× systemd-networkd-wait-online.service - Wait for Network to be Configured
     Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled-runtime; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sat 2023-11-04 15:45:37 CST; 50s ago
       Docs: man:systemd-networkd-wait-online.service(8)
    Process: 630 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
   Main PID: 630 (code=exited, status=1/FAILURE)
        CPU: 5ms

Nov 04 15:43:37 ubuntu systemd[1]: Starting Wait for Network to be Configured...
Nov 04 15:45:37 ubuntu systemd-networkd-wait-online[630]: Timeout occurred while waiting for network connectivity.
Nov 04 15:45:37 ubuntu systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 15:45:37 ubuntu systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Nov 04 15:45:37 ubuntu systemd[1]: Failed to start Wait for Network to be Configured.

root@ubuntu:~# SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 3
Found link 2
Found link 1
eth0: link is ignored
eth1: link is not managed by networkd (yet?).
lo: link is ignored
Timeout occurred while waiting for network connectivity.

[David Bürgin (glts)]

I think this bug should be reopened.

The timeout error still happens even after installing the stable release update.

[mev (mev1)]

I wish I was in the office as little as Herr Schmidt!

[Werner Harnisch (wharnisch2)]

This bug should be reopened.

The issue "A start job is running for Wait For Network to be Configured hanging at boot for 2 minutes in Ubuntu 22.04 servers" for me on Ubuntu 22.04 serverThe timeout error still happens even after installing the stable release update.

[Veli-Jussi Raitila (vjraitila)]

This is indeed still broken. Running on Raspberry Pi 4 ubuntu-server image.

# dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==================-============-=================================
ii systemd 249.11-0ubuntu3.11 arm64 system and service manager

# netplan get
network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true

# networkctl list
IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  3 wlan0 wlan off unmanaged

3 links listed.

# SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-networkd-wait-online --timeout=10
Found link 3
Found link 2
Found link 1
wlan0: link is not managed by networkd (yet?).
lo: link is ignored
eth0: link is ignored
Timeout occurred while waiting for network connectivity.

---

Is it by design for unmanaged interfaces to cause a timeout? Shouldn't wait-online exit immediately if all interfaces are optional OR unmanaged?

On the Raspberry Pi server image the default cloud-init network configuration only includes the eth0 interface. But e.g. Raspberry Pi 4 also has a WiFi chip, hence leaving the wlan0 interface unmanaged. This results in a timeout on a clean system.

The comment #52 seems to exhibit the same issue, but with eth1.

[Roger Nelson (rvnelson)]

I'm curious if this bug is still being investigated. RPI4 with wlan0, eth0 and eth1 (USB3 using r8152 driver) definitely manifests this bug. I need to wait ~30 minutes before either wlan0 or eth0 becomes available in order to restart systemd-networkd to activate eth1 and then ip link set up eth1. At that point the network appears stable until next reboot.

IDX LINK TYPE OPERATIONAL SETUP
  1 lo loopback carrier unmanaged
  2 eth0 ether routable configured
  4 wlan0 wlan routable unmanaged
  5 tun0 none routable unmanaged
  6 docker0 bridge routable unmanaged
  8 vethd334874 ether degraded unmanaged
  9 eth1 ether routable configured
 14 veth1a9ad92 ether degraded unmanaged
 16 vethe09f689 ether degraded unmanaged
 18 veth592971d ether degraded unmanaged
 19 macvlan-shim ether routable unmanaged
 21 vethafb615d ether degraded unmanaged

I hesitate using the word arbitrary but given how RPI4 utilizes both networkmanager and networkd it appears arbitrary to state that only networkmanager should be used in a desktop installation - at least I cannot find any documentation that supports not using both simultaneously. That said, I have been running this configuration for at least 2 years without issue *until* systemd was updated.

I'm a new Ubuntu-One user and this is my first post after waiting patiently for post #52 & #58 to get responses from anyone - so please forgive (and forget :). Constructive criticism humbly accepted.

[Steve Langasek (vorlon)]

On Sun, Dec 10, 2023 at 01:45:36AM -0000, Roger Nelson wrote:
> I'm curious if this bug is still being investigated. RPI4 with wlan0,
> eth0 and eth1 (USB3 using r8152 driver) definitely manifests this bug. I
> need to wait ~30 minutes before either wlan0 or eth0 becomes available
> in order to restart systemd-networkd to activate eth1 and then ip link
> set up eth1.

This bug report is about a change in behavior of the systemd wait-online
target at boot. "Need to wait 30 minutes before network device becomes
available" sounds unrelated. You should file a separate bug report for your
issue.

[Roger Nelson (rvnelson)]

Thank you Steve, perhaps you are right and unrelated. I will open a separate bug. What confused me (still) is why systemd-wait-online.service fails same/similar???

sudo systemctl status systemd-networkd-wait-online.service
× systemd-networkd-wait-online.service - Wait for Network to be Configured
     Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled-runtime; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sun 2023-12-10 07:51:38 PST; 11min ago
       Docs: man:systemd-networkd-wait-online.service(8)
    Process: 1096 ExecStart=/lib/systemd/systemd-networkd-wait-online --timeout=5 --any (code=exited, status=1/FAILURE)
   Main PID: 1096 (code=exited, status=1/FAILURE)
        CPU: 24ms

Dec 10 07:34:09 raspi2 systemd[1]: Starting Wait for Network to be Configured...
Dec 10 07:50:42 raspi2 systemd[1]: Dependency failed for Wait for Network to be Configured.
Dec 10 07:50:42 raspi2 systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
Dec 10 07:51:38 raspi2 systemd-networkd-wait-online[1096]: Timeout occurred while waiting for network connectivity.
Dec 10 07:51:38 raspi2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Dec 10 07:51:38 raspi2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.

[Veli-Jussi Raitila (vjraitila)]

I would not be surprised if your issue was at least related as you have a bunch of unconfigured interfaces. As far as I understand, the fix discussed here only addresses the case when an interface is marked as optional. However, the same timeout occurs also for UNMANAGED interfaces - a behavior which does not make sense to me. At least that is the case on my Raspberry Pi using the server image.

A cursory search around the internet came up with this https://github.com/systemd/systemd/issues/25813

[Roger Nelson (rvnelson)]

Thank you Veli-Jussi for your comment and providing this information. It appears this or similar problem has been around for a while but as I stated in my earlier post I only encountered this timeout issue after the update and it still exists in 249.11-0ubuntu3.11. After the dreaded timeout (and for USB3 dongle eth1 a restart of systemd-networkd) both eth0 and eth1 'networkct list' them as 'configured' as seen in previous post so unless I'm misreading this both interfaces are managed by networkd.

[Veli-Jussi Raitila (vjraitila)]

I encountered the same problem after an update as well.

The upstream issue I linked to, although quite old, discusses systemd version 252.3, whereas at least jammy has a patched version of 249.11 - which is even older.

And you do have e.g. wlan0 unmanaged as do I, which alone causes the timeout for me. There are ways to work around it, sure, but as it stands now the LTS server image for Raspberry Pi is effectively broken by default.

[Nick Rosbrook (enr0n)]

If you have a bunch of unmanaged interfaces, then you should use a drop-in configuration to pass the --any flag to systemd-networkd-wait-online:

cat > /etc/systemd/system/systemd-networkd-wait-online.service.d/any.conf << EOF
[Service]
ExecStart=
ExecStart=/lib/systemd/systemd-networkd-wait-online --any
EOF

This makes it so that systemd-networkd-wait-online will return success once at least one interface becomes configured.

[Roger Nelson (rvnelson)]

My shotgun approach may seem unorthodox but I've been frustrated since I am confident the systemd update created this issue. However, I withdraw my hesitation to use the word arbitrary. :D

Data point 1: Imaged a vanilla RPI4 with Ubuntu 22.04. The only thing I changed was the netplan config to mimic my ethernet config for eth0 and eth1 (USB3 dongle). Fails as before with systemd-networkd-wait-online.service failing, and ~30 minute timeout before network was available and able to ssh into the RPI4. Still eth1 USB dongle had to be systemd-networkd restart to obtain IP address.

Data point 2: do-release-upgrade of newly imaged RPI4 to 23.04. Still fails with systemd-networkd-wait-online.service failing but with only ~20 timeout and eth1 comes up with IP address after reboot.

Data point 3: do-release-upgrade of newly imaged RPI 23.04 to 23.10. Works flawlessly. No timeout. Systemd-networkd-wait-online.service no longer failing.

Data point 4: Performed the same do-release-upgrades on the original failing RPI4 and the same resulted on 23.04 and now 23.10. I now have my original RPI4 working like it was prior to the systemd update. I guess I should be stoked, and I am happy to be relieved of that dreaded timeout but I wanted to pass on this detail in case it sparked a level of interest. I can't imagine that I am the only one that has experienced this. And even if I am it is so curious. Best regards --any :D

[Veli-Jussi Raitila (vjraitila)]

So you eventually upgraded both raspis to 23.10? I find it hardly surprising it works as mantic has a newer upstream release of systemd 253.5 which has the fix I linked to above for ignoring unmanaged interfaces. Does not help for the LTS release (jammy) though.

[Roger Nelson (rvnelson)]

Indeed. I would have preferred to remain on LTS.

[Erwin (pierwin)]

I've reported the same reproducible issue on Raspberry Pi troubleshooting forum with Ubuntu 22.04.3 LTS 64-bit server as I was not aware of this bugreport:
https://forums.raspberrypi.com/viewtopic.php?p=2172165

> systemctl disable --now systemd-networkd.service

This does not work for me as the service is re-triggered by other cloud services.

Confirmed the following:
- Ubuntu 23.04 server 64-bit: Fixed, normal boottime
- Ubuntu 23.04.3 LTS server 64-bit with all updates: No solution?

Am I correct that there is still no solution for Raspberry Pi4 Ubuntu 22.04.3 LTS 64-bit server?

[Hans-Peter Schmidt (hps-ruetz)]

Guten Tag!

Ich bin am 02.01.2024 wieder im Büro.
Ihre Nachricht wird NICHT automatisch weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an Markus Delorenzo:
[phone]
md(@)ruetz.at

Mit freundlichen Grüßen
Hans-Peter Schmidt

[Steve Langasek (vorlon)]

There is no workaround for this with the stock netplan config on raspberry pi, which trips this bug. But you can customize /etc/netplan/ on your system to mark your expected/required network interfaces as "optional: false" (which is the default if unspecified); AIUI that would avoid this issue with systemd-networkd.

[Dave Jones (waveform)]

> If you have a bunch of unmanaged interfaces, then you should use a
> drop-in configuration to pass the --any flag to
> systemd-networkd-wait-online:
>
> cat > /etc/systemd/system/systemd-networkd-wait-online.service.d/any.conf << EOF
> [Service]
> ExecStart=
> ExecStart=/lib/systemd/systemd-networkd-wait-online --any
> EOF
>
> This makes it so that systemd-networkd-wait-online will return success once at least one interface becomes configured.

Unfortunately this doesn't appear to work. I've tried this on the
current noble dailies (which also suffer the 2 minute boot delay on
the server images due to this issue). The default netplan config is
simply:

  network:
    version: 2
    ethernets:
      eth0:
        dhcp4: true
        optional: true

This displays the issue, even with --any presumably because wlan0 is
unmanaged so networkctl reports the online state is "unknown" (even
when eth0 is happily configured and routable).

I attempting making wlan0 "managed" but optional. This requires a
mandatory access point to be defined to conform to the YAML schema, so
I added a fake one (I should open a separate issue for this because I
shouldn't have to make up access points just to mark an interface
optional):

  network:
    version: 2
    ethernets:
      eth0:
        dhcp4: true
        optional: true
    wifis:
      wlan0:
        dhcp4: true
        optional: true
        access-points:
          foo:
            password: foobarbaz

Unfortunately this also displays the issue. This time wlan0 is
managed, but "configuring" and the online state is still "unknown". So
far as I can tell, the --any option doesn't match the behaviour
claimed in the man-page (i.e. "exits with success when at least one
interface becomes online").

> There is no workaround for this with the stock netplan config on
> raspberry pi, which trips this bug. But you can customize
> /etc/netplan/ on your system to mark your expected/required network
> interfaces as "optional: false" (which is the default if
> unspecified); AIUI that would avoid this issue with
> systemd-networkd.

This *does* work ... assuming the Pi actually has a network interface.
If the Pi is networkless, it's unfortunately doomed to the 2 minute
delay still. And obviously I don't want to make eth0 mandatory by
default in the images (which would break the 3A+ and Zero 2W, which
lack eth0 entirely).

[Rahav Dor (rahavd)]

I have a number of RPi installed with Ubuntu Server 23.10 (64 bit) installations in Nov. 2023. They do no have this problem. Installing the same OS about a week ago on the same RPi, causes the problem. So the problem was likely caused by something Ubuntu released between Nov. to March.

@Steve Langasek [workaround](https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2036358/comments/74) works.

[Hans-Peter Schmidt (hps-ruetz)]

Guten Tag!

Ich bin am 22.04.2024 wieder im Büro.
Ihre Nachricht wird NICHT automatisch weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an Markus Delorenzo:
[phone]
md(@)ruetz.at

Mit freundlichen Grüßen
Hans-Peter Schmidt

[Laurent Bonnaud (laurent-bonnaud)]

Hi,
I got hit by this bug after a mantic -> noble upgrade on a desktop system.

[Hans-Peter Schmidt (hps-ruetz)]

Guten Tag!

Ich bin am 22.04.2024 wieder im Büro.
Ihre Nachricht wird NICHT automatisch weitergeleitet.

In dringenden Fällen wenden Sie sich bitte an Markus Delorenzo:
[phone]
md(@)ruetz.at

Mit freundlichen Grüßen
Hans-Peter Schmidt

[bam (mybigspam)]

Got it on mantic -> noble upgrade too, the solution proposed in the Description helped:

$ systemctl disable --now systemd-networkd.service