I looked into this, and the issue only happens if such interfaces (with state DOWN and NO-CARRIER) are managed by systemd-networkd (check with 'networkctl list').
Per systemd-networkd-wait-online.service man page [1]:
'By default, it will wait for all links it is aware of and which are managed by systemd-networkd.service(8) to be fully configured or failed,'
The issue can be reproduced in a VM with a NIC configured in libvirt XML as "<interface type='ethernet'><link state=down'/></interface>" that is managed by netplan. Steps below.
Check that systemd-networkd-wait-online.service is happy with 'ens3' only:
---
By default, only the 'ens3' interface is configured in netplan
(thus managed by systemd-networkd, default renderer in netplan).
$ ip -o l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\ link/ether 52:54:00:b8:ff:f1 brd ff:ff:ff:ff:ff:ff
3: ens7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000\ link/ether 52:54:00:fb:c6:b6 brd ff:ff:ff:ff:ff:ff
$ ls -1 /run/systemd/network/*.network
/run/systemd/network/10-netplan-ens3.network
/run/systemd/network/10-netplan-ens7.network
Now systemd-networkd-wait-online.service cares about it (SETUP status is 'configuring'):
$ sudo networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 ens3 ether routable configured
3 ens7 ether no-carrier configuring
3 links listed.
And sure enough, systemd-networkd-wait-online.service now times out:
$ time sudo systemctl restart systemd-networkd-wait-online.service
<... wait ...>
Job for systemd-networkd-wait-online.service failed because the control process exited with error code.
See "systemctl status systemd-networkd-wait-online.service" and "journalctl -xe" for details.
real 2m0.260s
user 0m0.012s
sys 0m0.011s
$ systemctl status systemd-networkd-wait-online.service
● systemd-networkd-wait-online.service - Wait for Network to be Configured
...
Process: 1215 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 1215 (code=exited, status=1/FAILURE)
...
Feb 12 00:01:08 bionic systemd-networkd-wait-online[1215]: Event loop failed: Connection timed out
Feb 12 00:01:08 bionic systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Feb 12 00:01:08 bionic systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Feb 12 00:01:08 bionic systemd[1]: Failed to start Wait for Network to be Configured.
Also, notice that the NO-CARRIER bit only showed up now, presumably after systemd-networkd set the bit UP:
$ ip -o l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\ link/ether 52:54:00:b8:ff:f1 brd ff:ff:ff:ff:ff:ff
3: ens7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000\ link/ether 52:54:00:fb:c6:b6 brd ff:ff:ff:ff:ff:ff
ubuntu@bionic:~$
Hi Ed,
I looked into this, and the issue only happens if such interfaces (with state DOWN and NO-CARRIER) are managed by systemd-networkd (check with 'networkctl list').
Per systemd- networkd- wait-online. service man page [1]:
'By default, it will wait for all links it is aware of and which are managed by systemd- networkd. service( 8) to be fully configured or failed,'
[1] https:/ /www.freedeskto p.org/software/ systemd/ man/systemd- networkd- wait-online. service. html
The issue can be reproduced in a VM with a NIC configured in libvirt XML as "<interface type='ethernet' ><link state=down' /></interface> " that is managed by netplan. Steps below.
Hope this helps,
Mauricio
---
Create a VM (bionic):
---
$ uvt-simplestrea ms-libvirt sync release=bionic arch=amd64
$ uvt-kvm create --cpu 2 --memory 2048 --disk 4 --password password bionic release=bionic arch=amd64
$ uvt-kvm wait bionic
Give it an ethernet interface with link down:
---
$ virsh edit bionic
...
<interface type='ethernet'>
<model type='virtio'/>
<link state='down'/>
</interface>
...
Re-start the VM:
---
$ virsh shutdown bionic
$ virsh start bionic
$ uvt-kvm wait bionic
$ uvt-kvm ssh bionic
Check that systemd- networkd- wait-online. service is happy with 'ens3' only:
---
By default, only the 'ens3' interface is configured in netplan
(thus managed by systemd-networkd, default renderer in netplan).
$ ip -o l UP,LOWER_ UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 MULTICAST, UP,LOWER_ UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\ link/ether 52:54:00:b8:ff:f1 brd ff:ff:ff:ff:ff:ff MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000\ link/ether 52:54:00:fb:c6:b6 brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,
2: ens3: <BROADCAST,
3: ens7: <BROADCAST,
$ grep -r ens[0-9]: /etc/netplan 50-cloud- init.yaml: ens3:
/etc/netplan/
$ ls -1 /run/systemd/ network/ *.network network/ 10-netplan- ens3.network
/run/systemd/
Notice that the 'ens7' interface (other/new) SETUP status is 'unmanaged':
$ sudo networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 ens3 ether routable configured
3 ens7 ether off unmanaged
3 links listed.
Thus, despite being 'state DOWN', systemd- networkd- wait-online. service doesn't care, and succeeds:
$ systemctl status systemd- networkd- wait-online. service | grep Process /lib/systemd/ systemd- networkd- wait-online (code=exited, status=0/SUCCESS)
Process: 611 ExecStart=
Check that systemd- networkd- wait-online. service gets unhappy with 'ens7' too:
---
If you just configure 'ens7' in netplan, even without setting/getting IP:
$ cat <<EOF | sudo tee /etc/netplan/ ens7.yaml
network:
ethernets:
ens7:
match:
macaddress: 52:54:00:fb:c6:b6
set-name: ens7
version: 2
EOF
$ sudo netplan apply
$ grep -r ens[0-9]: /etc/netplan ens7.yaml: ens7: 50-cloud- init.yaml: ens3:
/etc/netplan/
/etc/netplan/
$ ls -1 /run/systemd/ network/ *.network network/ 10-netplan- ens3.network network/ 10-netplan- ens7.network
/run/systemd/
/run/systemd/
Now systemd- networkd- wait-online. service cares about it (SETUP status is 'configuring'):
$ sudo networkctl list
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier unmanaged
2 ens3 ether routable configured
3 ens7 ether no-carrier configuring
3 links listed.
And sure enough, systemd- networkd- wait-online. service now times out:
$ time sudo systemctl restart systemd- networkd- wait-online. service networkd- wait-online. service failed because the control process exited with error code. networkd- wait-online. service" and "journalctl -xe" for details.
<... wait ...>
Job for systemd-
See "systemctl status systemd-
real 2m0.260s
user 0m0.012s
sys 0m0.011s
$ systemctl status systemd- networkd- wait-online. service networkd- wait-online. service - Wait for Network to be Configured /lib/systemd/ systemd- networkd- wait-online (code=exited, status=1/FAILURE) networkd- wait-online[ 1215]: Event loop failed: Connection timed out networkd- wait-online. service: Main process exited, code=exited, status=1/FAILURE networkd- wait-online. service: Failed with result 'exit-code'.
● systemd-
...
Process: 1215 ExecStart=
Main PID: 1215 (code=exited, status=1/FAILURE)
...
Feb 12 00:01:08 bionic systemd-
Feb 12 00:01:08 bionic systemd[1]: systemd-
Feb 12 00:01:08 bionic systemd[1]: systemd-
Feb 12 00:01:08 bionic systemd[1]: Failed to start Wait for Network to be Configured.
Also, notice that the NO-CARRIER bit only showed up now, presumably after systemd-networkd set the bit UP:
$ ip -o l UP,LOWER_ UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 MULTICAST, UP,LOWER_ UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000\ link/ether 52:54:00:b8:ff:f1 brd ff:ff:ff:ff:ff:ff BROADCAST, MULTICAST, UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000\ link/ether 52:54:00:fb:c6:b6 brd ff:ff:ff:ff:ff:ff
1: lo: <LOOPBACK,
2: ens3: <BROADCAST,
3: ens7: <NO-CARRIER,
ubuntu@bionic:~$