networkd-dispatcher starts too late to take effect on boot

[Steve Langasek]

[Impact]
networkd-dispatcher's systemd unit has default dependencies, which means it starts much, much too late to have any effect for interfaces brought up by networkd at boot (since the events are missed, and not replayed).

[Test Case]
Start systemd-networkd / boot system with it, ensure the current status is replayed when networkd-dispatcher starts.

[Regression potential]
The current status is replayed when the service is restarted, and more events run at boot, potentially causing slowdown/problems with hooks not expecting to receive an event a second time (after restart). That said, not replaying the current status on restart would mean we might lose important events during that window.

[Original approach]
We should instead have systemd unit dependencies like this (replicating systemd-networkd's own):

[Unit]
Description=Dispatcher daemon for systemd-networkd
DefaultDependencies=no
After=systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=systemd-networkd.service

In local testing, this DTRT.

[Julian Andres Klode (juliank)]

We probably also should replace WantedBy=multi-user.target with WantedBy=systemd-networkd.service so we only start it if systemd-networkd is started. Or add an Also=networkd-dispatcher.service to systemd-networkd.service; I don't know what I prefer.

[Julian Andres Klode (juliank)]

Forwarded to https://github.com/craftyguy/networkd-dispatcher/issues/40, fixed in https://github.com/craftyguy/networkd-dispatcher/commit/87e5541c6039ab52c12031371147e8687a37bd10 - upstream also added an After=dbus.service (maybe it should be Wants/After=dbus.socket, though?)

[Launchpad Janitor (janitor)]

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu4

---------------
networkd-dispatcher (1.7-0ubuntu4) cosmic; urgency=medium

  * Cherry-pick upstream fix to start before systemd-networkd (LP: #1772137)

 -- Julian Andres Klode <email address hidden> Fri, 25 May 2018 16:08:00 +0200

[Scott Moser (smoser)]

I believe that this change has broken cosmic cloud-images.
there is 'journalctl -o short-precise' output of a 20180530 image booted in kvm at
 http://paste.ubuntu.com/p/xhzxNc4QtR/

From that you will see:
  May 31 15:54:35.838996 ubuntu systemd[1]: sysinit.target: Found ordering cycle on cloud-init.service/start
  May 31 15:54:35.839017 ubuntu systemd[1]: sysinit.target: Found dependency on systemd-networkd-wait-online.service/start
  May 31 15:54:35.839042 ubuntu systemd[1]: sysinit.target: Found dependency on systemd-networkd.service/start
  May 31 15:54:35.839064 ubuntu systemd[1]: sysinit.target: Found dependency on networkd-dispatcher.service/start

The result is that cloud-init.service does not run as it needs to.

[Brian Murray (brian-murray)]

I'm going to hold off on approving the SRU until the cosmic issue has been investigated.

[Julian Andres Klode (juliank)]

Reverted the change for now, this needs more work. I guess we should reject the bionic SRU and figure out what to do.

[Launchpad Janitor (janitor)]

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu6

---------------
networkd-dispatcher (1.7-0ubuntu6) cosmic; urgency=medium

  * networkd-dispatcher.conf: Run startup triggers by default (LP: #1772137)

 -- Julian Andres Klode <email address hidden> Tue, 19 Jun 2018 11:19:39 +0200

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted networkd-dispatcher into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/networkd-dispatcher/1.7-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Julian Andres Klode (juliank)]

I added an example hook for routable, and started networkd-dispatcher on the system:

with -3:
Jun 25 11:08:58 bionic systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 25 11:08:58 bionic networkd-dispatcher[4776]: No valid path found for iwconfig
Jun 25 11:08:58 bionic networkd-dispatcher[4776]: No valid path found for iw
Jun 25 11:08:58 bionic systemd[1]: Started Dispatcher daemon for systemd-networkd.

with -3.2:

Jun 25 11:07:50 bionic systemd[1]: Starting Dispatcher daemon for systemd-networkd...
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: No valid path found for iwconfig
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: No valid path found for iw
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export ADDR='10.33.102.18'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export AdministrativeState='configured'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export ESSID=''
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IFACE='eth0'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export INVOCATION_ID='20a589ee6163485da884a8804adccb9c'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IP6_ADDRS='fd42:38f6:7979:e683:216:3eff:fedd:a175'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export IP_ADDRS='10.33.102.18'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export JOURNAL_STREAM='9:38040655'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export LANG='C.UTF-8'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export NOTIFY_SOCKET='/run/systemd/notify'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export OperationalState='routable'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export PWD='/'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export STATE='routable'
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export json='{"Type": "ether", "OperationalState": "routable", "AdministrativeState": "confi
Jun 25 11:07:50 bionic networkd-dispatcher[4622]: export networkd_dispatcher_args='--run-startup-triggers'
Jun 25 11:07:50 bionic systemd[1]: Started Dispatcher daemon for systemd-networkd.

=> Seems to work

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for networkd-dispatcher has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package networkd-dispatcher - 1.7-0ubuntu3.2

---------------
networkd-dispatcher (1.7-0ubuntu3.2) bionic; urgency=medium

  * Revert "Cherry-pick upstream fix to start before systemd-networkd"
  * networkd-dispatcher.conf: Run startup triggers by default (LP: #1772137)

networkd-dispatcher (1.7-0ubuntu3.1) bionic; urgency=medium

  * Cherry-pick upstream fix to start before systemd-networkd (LP: #1772137)
  * debian/gbp.conf: Point to ubuntu/bionic branch

 -- Julian Andres Klode <email address hidden> Tue, 19 Jun 2018 12:03:47 +0200

[Sebastian Unger (sebunger44)]

It appears this has been "unfixed" in focal?

root@ttde-kvm6:~# systemctl cat networkd-dispatcher.service
# /lib/systemd/system/networkd-dispatcher.service
[Unit]
Description=Dispatcher daemon for systemd-networkd

[Service]
Type=notify
ExecStart=/usr/bin/networkd-dispatcher $networkd_dispatcher_args

# Load /etc/default/networkd-dispatcher
EnvironmentFile=-/etc/default/%p

[Install]
WantedBy=multi-user.target

And dispatcher does not run scripts on start-up.

[Sebastian Unger (sebunger44)]

Never mind. The scripts were in the wrong directory. /lib instead of /usr/lib.