networkd-dispatcher starts too late to take effect on boot

Bug #1772137 reported by Steve Langasek on 2018-05-19
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
networkd-dispatcher (Ubuntu)
High
Julian Andres Klode
Bionic
High
Unassigned

Bug Description

[Impact]
networkd-dispatcher's systemd unit has default dependencies, which means it starts much, much too late to have any effect for interfaces brought up by networkd at boot (since the events are missed, and not replayed).

[Test Case]
Start systemd-networkd / boot system with it, ensure the current status is replayed when networkd-dispatcher starts.

[Regression potential]
The current status is replayed when the service is restarted, and more events run at boot, potentially causing slowdown/problems with hooks not expecting to receive an event a second time (after restart). That said, not replaying the current status on restart would mean we might lose important events during that window.

[Original approach]
We should instead have systemd unit dependencies like this (replicating systemd-networkd's own):

[Unit]
Description=Dispatcher daemon for systemd-networkd
DefaultDependencies=no
After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=systemd-networkd.service

In local testing, this DTRT.

Steve Langasek (vorlon) on 2018-05-19
Changed in networkd-dispatcher (Ubuntu):
assignee: nobody → Julian Andres Klode (juliank)
importance: Undecided → High
Revision history for this message
Julian Andres Klode (juliank) wrote :

We probably also should replace WantedBy=multi-user.target with WantedBy=systemd-networkd.service so we only start it if systemd-networkd is started. Or add an Also=networkd-dispatcher.service to systemd-networkd.service; I don't know what I prefer.

Changed in networkd-dispatcher (Ubuntu):
status: New → In Progress
Changed in networkd-dispatcher (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Julian Andres Klode (juliank) wrote :

Forwarded to https://github.com/craftyguy/networkd-dispatcher/issues/40, fixed in https://github.com/craftyguy/networkd-dispatcher/commit/87e5541c6039ab52c12031371147e8687a37bd10 - upstream also added an After=dbus.service (maybe it should be Wants/After=dbus.socket, though?)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu4

---------------
networkd-dispatcher (1.7-0ubuntu4) cosmic; urgency=medium

  * Cherry-pick upstream fix to start before systemd-networkd (LP: #1772137)

 -- Julian Andres Klode <email address hidden> Fri, 25 May 2018 16:08:00 +0200

Changed in networkd-dispatcher (Ubuntu):
status: In Progress → Fix Released
tags: added: id-5affa8cf4534263e12fd1d5b
description: updated
Changed in networkd-dispatcher (Ubuntu Bionic):
status: Triaged → In Progress
Revision history for this message
Scott Moser (smoser) wrote :

I believe that this change has broken cosmic cloud-images.
there is 'journalctl -o short-precise' output of a 20180530 image booted in kvm at
 http://paste.ubuntu.com/p/xhzxNc4QtR/

From that you will see:
  May 31 15:54:35.838996 ubuntu systemd[1]: sysinit.target: Found ordering cycle on cloud-init.service/start
  May 31 15:54:35.839017 ubuntu systemd[1]: sysinit.target: Found dependency on systemd-networkd-wait-online.service/start
  May 31 15:54:35.839042 ubuntu systemd[1]: sysinit.target: Found dependency on systemd-networkd.service/start
  May 31 15:54:35.839064 ubuntu systemd[1]: sysinit.target: Found dependency on networkd-dispatcher.service/start

The result is that cloud-init.service does not run as it needs to.

description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

I'm going to hold off on approving the SRU until the cosmic issue has been investigated.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Reverted the change for now, this needs more work. I guess we should reject the bionic SRU and figure out what to do.

Changed in networkd-dispatcher (Ubuntu Bionic):
status: In Progress → Confirmed
status: Confirmed → Triaged
Changed in networkd-dispatcher (Ubuntu):
status: Fix Released → Triaged
status: Triaged → In Progress
tags: added: id-5b1030780cc11291cbe073af
Changed in networkd-dispatcher (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu6

---------------
networkd-dispatcher (1.7-0ubuntu6) cosmic; urgency=medium

  * networkd-dispatcher.conf: Run startup triggers by default (LP: #1772137)

 -- Julian Andres Klode <email address hidden> Tue, 19 Jun 2018 11:19:39 +0200

Changed in networkd-dispatcher (Ubuntu):
status: Fix Committed → Fix Released
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Steve, or anyone else affected,

Accepted networkd-dispatcher into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/networkd-dispatcher/1.7-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in networkd-dispatcher (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Julian Andres Klode (juliank) wrote :

I added an example hook for routable, and started networkd-dispatcher on the system:

with -3:
Jun 25 11:08:58 bionic systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 25 11:08:58 bionic networkd-dispatcher[4776]: No valid path found for iwconfig
Jun 25 11:08:58 bionic networkd-dispatcher[4776]: No valid path found for iw
Jun 25 11:08:58 bionic systemd[1]: Started Dispatcher daemon for systemd-networkd.

with -3.2:

Jun 25 11:07:50 bionic systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: No valid path found for iwconfig
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: No valid path found for iw
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export ADDR='10.33.102.18'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export AdministrativeState='configured'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export ESSID=''
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IFACE='eth0'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export INVOCATION_ID='20a589ee6163485da884a8804adccb9c'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IP6_ADDRS='fd42:38f6:7979:e683:216:3eff:fedd:a175'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IP_ADDRS='10.33.102.18'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export JOURNAL_STREAM='9:38040655'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export LANG='C.UTF-8'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export NOTIFY_SOCKET='/run/systemd/notify'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export OperationalState='routable'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export PWD='/'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export STATE='routable'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export json='{"Type": "ether", "OperationalState": "routable", "AdministrativeState": "confi
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export networkd_dispatcher_args='--run-startup-triggers'
Jun 25 11:07:50 bionic systemd[1]: Started Dispatcher daemon for systemd-networkd.

=> Seems to work

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for networkd-dispatcher has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu3.2

---------------
networkd-dispatcher (1.7-0ubuntu3.2) bionic; urgency=medium

  * Revert "Cherry-pick upstream fix to start before systemd-networkd"
  * networkd-dispatcher.conf: Run startup triggers by default (LP: #1772137)

networkd-dispatcher (1.7-0ubuntu3.1) bionic; urgency=medium

  * Cherry-pick upstream fix to start before systemd-networkd (LP: #1772137)
  * debian/gbp.conf: Point to ubuntu/bionic branch

 -- Julian Andres Klode <email address hidden> Tue, 19 Jun 2018 12:03:47 +0200

Changed in networkd-dispatcher (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Sebastian Unger (sebunger44) wrote :

It appears this has been "unfixed" in focal?

root@ttde-kvm6:~# systemctl cat networkd-dispatcher.service
# /lib/systemd/system/networkd-dispatcher.service
[Unit]
Description=Dispatcher daemon for systemd-networkd

[Service]
Type=notify
ExecStart=/usr/bin/networkd-dispatcher $networkd_dispatcher_args

# Load /etc/default/networkd-dispatcher
EnvironmentFile=-/etc/default/%p

[Install]
WantedBy=multi-user.target

And dispatcher does not run scripts on start-up.

Revision history for this message
Sebastian Unger (sebunger44) wrote :

Never mind. The scripts were in the wrong directory. /lib instead of /usr/lib.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.