| 2018-11-21 15:44:32 |
Mikko Korkalo |
bug |
|
|
added bug |
| 2018-12-03 12:27:29 |
Benjamin Drung |
summary |
netplan dhcp interface with no default route causes causes systemd-networkd-wait-online to hang |
netplan dhcp interface with no default route causes systemd-networkd-wait-online to hang |
|
| 2018-12-03 12:27:33 |
Launchpad Janitor |
systemd (Ubuntu): status |
New |
Confirmed |
|
| 2018-12-03 12:30:43 |
Benjamin Drung |
bug watch added |
|
https://github.com/systemd/systemd/issues/3752 |
|
| 2018-12-03 12:30:43 |
Benjamin Drung |
bug task added |
|
systemd |
|
| 2018-12-03 12:48:31 |
Juho K. Juopperi |
bug |
|
|
added subscriber Juho K. Juopperi |
| 2018-12-03 12:49:46 |
Benjamin Drung |
bug task deleted |
systemd |
|
|
| 2018-12-03 12:50:33 |
Benjamin Drung |
bug task added |
|
systemd |
|
| 2018-12-03 13:10:44 |
Benjamin Drung |
bug |
|
|
added subscriber Benjamin Drung |
| 2018-12-03 13:11:33 |
Benjamin Drung |
systemd: status |
New |
Fix Released |
|
| 2018-12-03 13:51:06 |
Benjamin Drung |
description |
root cause in systemd bug:
https://github.com/systemd/systemd/issues/3752
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
root cause in systemd bug fixed in Commit 223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2018-12-03 13:51:14 |
Benjamin Drung |
description |
root cause in systemd bug fixed in Commit 223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2018-12-03 13:51:27 |
Benjamin Drung |
systemd (Ubuntu): status |
Confirmed |
Triaged |
|
| 2019-04-08 22:26:09 |
dann frazier |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-04-08 22:26:09 |
dann frazier |
bug task added |
|
systemd (Ubuntu Bionic) |
|
| 2019-04-08 22:26:18 |
dann frazier |
systemd (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2019-04-09 00:05:14 |
Dan Streetman |
description |
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface, if also configured to ignore the dhcp server's route (or if the server provides no route), systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
[test case]
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 593 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 593 (code=exited, status=1/FAILURE)
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2019-04-09 00:05:19 |
Dan Streetman |
systemd (Ubuntu Bionic): assignee |
|
Dan Streetman (ddstreet) |
|
| 2019-04-09 00:05:21 |
Dan Streetman |
systemd (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2019-04-09 00:05:24 |
Dan Streetman |
systemd (Ubuntu Bionic): status |
Confirmed |
In Progress |
|
| 2019-04-09 00:05:31 |
Dan Streetman |
systemd (Ubuntu): status |
Triaged |
Fix Released |
|
| 2019-04-09 00:19:48 |
Dan Streetman |
bug |
|
|
added subscriber Dan Streetman |
| 2019-04-09 07:18:36 |
Christian Ehrhardt |
description |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface, if also configured to ignore the dhcp server's route (or if the server provides no route), systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
[test case]
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 593 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 593 (code=exited, status=1/FAILURE)
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
The fix is backporting upstream commit [1]
[1]: https://github.com/systemd/systemd/commit/223932c7
[test case]
There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=disco
$ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily
Create a isolated network:
$ cat > isolate.xml << EOF
<network>
<name>isolated</name>
<!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
<ip address='192.168.251.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.251.2' end='192.168.251.254'/>
</dhcp>
</ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolate.xml
Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
$ virsh edit bionic
#add this:
<interface type='network'>
<source network='isolated'/>
<mac address='52:54:00:c1:69:08'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>
In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
ens7:
dhcp4: true
match:
macaddress: 52:54:00:c1:69:08
set-name: ens7
$ sudo netplan apply
When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 563 (code=exited, status=1/FAILURE)
Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.
When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
The problem is that without a route there will be no netlink even t that can be tapped.
But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero).
Therefore the change seems rather small, reviewable and safe.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2019-04-09 07:20:22 |
Christian Ehrhardt |
bug |
|
|
added subscriber Christian Ehrhardt |
| 2019-04-16 09:22:38 |
Łukasz Zemczak |
systemd (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-04-16 09:22:40 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-04-16 09:22:42 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2019-04-16 09:22:46 |
Łukasz Zemczak |
tags |
systemd |
systemd verification-needed verification-needed-bionic |
|
| 2019-04-16 13:47:28 |
Christian Ehrhardt |
description |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
The fix is backporting upstream commit [1]
[1]: https://github.com/systemd/systemd/commit/223932c7
[test case]
There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=disco
$ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily
Create a isolated network:
$ cat > isolate.xml << EOF
<network>
<name>isolated</name>
<!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
<ip address='192.168.251.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.251.2' end='192.168.251.254'/>
</dhcp>
</ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolate.xml
Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
$ virsh edit bionic
#add this:
<interface type='network'>
<source network='isolated'/>
<mac address='52:54:00:c1:69:08'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>
In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
ens7:
dhcp4: true
match:
macaddress: 52:54:00:c1:69:08
set-name: ens7
$ sudo netplan apply
When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 563 (code=exited, status=1/FAILURE)
Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.
When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
The problem is that without a route there will be no netlink even t that can be tapped.
But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero).
Therefore the change seems rather small, reviewable and safe.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
The fix is backporting upstream commit [1]
[1]: https://github.com/systemd/systemd/commit/223932c7
[test case]
There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
$ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily
Create a isolated network:
$ cat > isolate.xml << EOF
<network>
<name>isolated</name>
<!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
<ip address='192.168.251.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.251.2' end='192.168.251.254'/>
</dhcp>
</ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolate
Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
$ virsh edit bionic
#add this:
<interface type='network'>
<source network='isolated'/>
<mac address='52:54:00:c1:69:08'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>
In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
ens7:
dhcp4: true
match:
macaddress: 52:54:00:c1:69:08
set-name: ens7
$ sudo netplan apply
When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 563 (code=exited, status=1/FAILURE)
Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.
When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
The problem is that without a route there will be no netlink even t that can be tapped.
But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero).
Therefore the change seems rather small, reviewable and safe.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2019-04-16 13:48:55 |
Christian Ehrhardt |
description |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
The fix is backporting upstream commit [1]
[1]: https://github.com/systemd/systemd/commit/223932c7
[test case]
There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
$ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily
Create a isolated network:
$ cat > isolate.xml << EOF
<network>
<name>isolated</name>
<!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
<ip address='192.168.251.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.251.2' end='192.168.251.254'/>
</dhcp>
</ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolate
Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
$ virsh edit bionic
#add this:
<interface type='network'>
<source network='isolated'/>
<mac address='52:54:00:c1:69:08'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>
In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
ens7:
dhcp4: true
match:
macaddress: 52:54:00:c1:69:08
set-name: ens7
$ sudo netplan apply
When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 563 (code=exited, status=1/FAILURE)
Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.
When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
The problem is that without a route there will be no netlink even t that can be tapped.
But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero).
Therefore the change seems rather small, reviewable and safe.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
[impact]
systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases:
a) if configured to ignore the dhcp server's route
b) the server provides no route
Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out).
This delays boot as well as any restart to systemd-networkd.
The fix is backporting upstream commit [1]
[1]: https://github.com/systemd/systemd/commit/223932c7
[test case]
There are two ways to test this.
A) make the system ignore Routes (slightly less realistic but easier to test)
configure an interface using systemd-networkd:
$ cat /etc/systemd/network/20-ens7.network
[Match]
Name=ens7
[Network]
DHCP=ipv4
[DHCP]
UseRoutes=false
then reboot, and check:
$ systemctl status systemd-networkd-wait-online
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago
[...]
Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured...
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3
Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo
Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured.
B) Make a dhcp server to not provide rules
Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
$ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily
Create a isolated network:
$ cat > isolate.xml << EOF
<network>
<name>isolated</name>
<!-- <bridge name='virbriso1' stp='on' delay='0'/> -->
<ip address='192.168.251.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.251.2' end='192.168.251.254'/>
</dhcp>
</ip>
</network>
EOF
$ virsh net-define isolate.xml
$ virsh net-start isolated
Edit the guest and add that network:
$ virsh shutdown bionic
$ virsh edit bionic
#add this:
<interface type='network'>
<source network='isolated'/>
<mac address='52:54:00:c1:69:08'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</interface>
In the guest make the device use DHCP:
Add this to /etc/netplan/50-cloud-init.yaml
ens7:
dhcp4: true
match:
macaddress: 52:54:00:c1:69:08
set-name: ens7
$ sudo netplan apply
When rebooting the guest again it will fail
● systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago
Docs: man:systemd-networkd-wait-online.service(8)
Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE)
Main PID: 563 (code=exited, status=1/FAILURE)
Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured...
Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3
Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'.
Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured.
When working cases A and B will both have the service starting fast and happy.
$ sudo systemctl status systemd-networkd-wait-online
Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3
[...]
Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7
[regression potential]
this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting.
The problem is that without a route there will be no netlink even t that can be tapped.
But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero).
Therefore the change seems rather small, reviewable and safe.
[other info]
original description:
--
root cause in systemd bug fixed in Commit
223932c786ada7f758a7b7878a6ad2dae0d1e5fb:
https://github.com/systemd/systemd/pull/8728
Environment:
Ubuntu 18.04 LTS amd64
systemd package version 237-3ubuntu10.9
How to trigger:
1. add netplan interface with dhcpv4 client enabled:
enp0s8:
addresses: []
dhcp4: true
2. configure dhcp server to NOT give a default route, or any route for that matter
3. reboot
4. systemd-networkd-wait-online will block until dhcp renew is triggered
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3
Event loop failed: Connection timed out
root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8
managing: enp0s3
ignoring: lo
ignoring: enp0s8
How to fix:
Backport upstream changes from systemd (see related systemd ticket). |
|
| 2019-04-16 17:25:15 |
dann frazier |
tags |
systemd verification-needed verification-needed-bionic |
systemd verification-done verification-done-bionic |
|
| 2019-04-17 03:37:19 |
Mathew Hodson |
systemd: importance |
Undecided |
Unknown |
|
| 2019-04-17 03:37:19 |
Mathew Hodson |
systemd: status |
Fix Released |
Unknown |
|
| 2019-04-17 03:37:19 |
Mathew Hodson |
systemd: remote watch |
|
github.com/systemd/systemd/issues #3752 |
|
| 2019-04-17 03:42:59 |
Mathew Hodson |
affects |
systemd |
ubuntu-translations |
|
| 2019-04-17 03:42:59 |
Mathew Hodson |
ubuntu-translations: importance |
Unknown |
Undecided |
|
| 2019-04-17 03:42:59 |
Mathew Hodson |
ubuntu-translations: status |
Unknown |
New |
|
| 2019-04-17 03:42:59 |
Mathew Hodson |
ubuntu-translations: remote watch |
github.com/systemd/systemd/issues #3752 |
|
|
| 2019-04-17 03:43:09 |
Mathew Hodson |
bug task deleted |
ubuntu-translations |
|
|
| 2019-04-17 03:43:16 |
Mathew Hodson |
systemd (Ubuntu): importance |
Undecided |
Medium |
|
| 2019-04-17 03:43:24 |
Mathew Hodson |
bug watch removed |
https://github.com/systemd/systemd/issues/3752 |
|
|
| 2019-04-23 09:35:23 |
Launchpad Janitor |
systemd (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-04-23 09:35:31 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2020-01-18 20:16:55 |
Dan Streetman |
removed subscriber Dan Streetman |
|
|
|
