dhclient triggers systemd-resolved start limit when processing more than 5 interfaces at once

[Dan Streetman]

[impact]

on a system where systemd-resolved is running, if dhclient is used on more than 5 interfaces, it calls the '/etc/dhcp/dhclient-enter-hooks.d/resolved' script multiple times, which then restarts systemd-resolved multiple times, triggering the unit's start-limit throttle which results in the unit entering failed state

[test case]

on a system with more than 5 available interfaces to run dhclient on (where all the interfaces will get a dhcp response), run dhclient with the interfaces:

$ dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15

check if systemd-resolved failed:

$ journalctl -b -1 -u systemd-resolved
...
Aug 09 00:38:08 sf316232-b systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Aug 09 00:38:08 sf316232-b systemd[1]: systemd-resolved.service: Failed with result 'start-limit-hit'.
Aug 09 00:38:08 sf316232-b systemd[1]: Failed to start Network Name Resolution.

[regression potential]

failure to start/stop/restart systemd-resolved, or problems adding dhclient-provided DNS nameservers to systemd-resolved

[scope]

this is needed only for b/f

the dhclient 'hook' script is provided by the systemd package in focal and earlier, and needs fixing in those releases

in h and later, the 'hook' script is included in the isc-dhcp-client package and notifies systemd-resolved in a more direct way without requiring restarting, and so doesn't trigger the restart limiting.

[Chris Newcomer (cnewcomer)]

I have tested this on both Focal and Bionic VMs. I ensured that the systemd-resolved unit was in a failed state, restarted it, added the PPA, updated the systemd packages, and rebooted. After every reboot the systemd-resolved was in an "active" state.

I tested this with at least 10 reboots on each affected Ubuntu release (b and f).

[Chris Halse Rogers (raof)]

Hello Dan, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.52 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.52) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

casync/2+61.20180112-1 (s390x)
nftables/0.8.2-1 (amd64)
lxc/3.0.3-0ubuntu1~18.04.1 (amd64, i386, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

Hello Dan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.12) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

flatpak/1.6.5-0ubuntu0.3 (amd64)
gvfs/1.44.1-1ubuntu1 (amd64, arm64)
munin/2.0.56-1ubuntu1 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

ubuntu@lp1939255-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.11 amd64 system and service manager
ubuntu@lp1939255-f:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-f:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:35:56 UTC, end at Sun 2021-09-05 21:40:13 UTC. --
...
Sep 05 21:40:13 lp1939255-f systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Sep 05 21:40:13 lp1939255-f systemd[1]: systemd-resolved.service: Failed with result 'start-limit-hit'.
Sep 05 21:40:13 lp1939255-f systemd[1]: Failed to start Network Name Resolution.

ubuntu@lp1939255-f:~$ dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.12 amd64 system and service manager
ubuntu@lp1939255-f:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-f:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:35:56 UTC, end at Sun 2021-09-05 21:44:25 UTC. --
...
Sep 05 21:44:23 lp1939255-f systemd[1]: Started Network Name Resolution.

[Dan Streetman (ddstreet)]

ubuntu@lp1939255-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.51 amd64 system and service manager
ubuntu@lp1939255-b:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-b:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:36:12 UTC, end at Sun 2021-09-05 21:41:06 UTC. --
...
Sep 05 21:40:57 lp1939255-b systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Sep 05 21:40:57 lp1939255-b systemd[1]: systemd-resolved.service: Failed with result 'start-limit-hit'.
Sep 05 21:40:57 lp1939255-b systemd[1]: Failed to start Network Name Resolution.

ubuntu@lp1939255-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.52 amd64 system and service manager
ubuntu@lp1939255-b:~$ sudo dhclient ens8 ens9 ens10 ens11 ens12 ens13 ens14 ens15
ubuntu@lp1939255-b:~$ journalctl --no-pager -b -u systemd-resolved
-- Logs begin at Sun 2021-09-05 21:36:12 UTC, end at Sun 2021-09-05 21:50:57 UTC. --
...
Sep 05 21:50:57 lp1939255-b systemd[1]: Started Network Name Resolution.

[Brian Murray (brian-murray)]

Hello Dan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.13) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.44.1-1ubuntu1 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

as systemd was respun (for focal) due to bug 1942899, and only the one udev (hwdb) patch was reverted which shouldn't affect this at all, I'm remarking this as verified still based on testing for the previous version above.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.52

---------------
systemd (237-3ubuntu10.52) bionic; urgency=medium

  * d/extra/dhclient-enter-resolved-hook:
    Reset start limit counter for systemd-resolved in dhclient hook
    (LP: #1939255)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ea6710476dde78e8595274c3c4ba7acca6d5162c
  * d/p/lp1934147/0001-core-add-a-new-unit-method-catchup.patch,
    d/p/lp1934147/0002-cgroup-do-catchup-for-unit-cgroup-inotify-watch-file.patch,
    d/p/lp1934147/0003-core-Make-sure-cgroup_oom_queue-is-flushed-on-manage.patch:
    Catch up on cgroup empty inotify after reexec/reload (LP: #1934147)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5ef61bd930612a90ce3ed9105cbadc5ff97b6ffc
  * d/p/lp1934981-correct-suspend-then-sleep-string.patch:
    Fix sleep verb used by logind during suspend-then-hibernate
    (LP: #1934981)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1ade873a41ad018a5e07f10775738c6eb8c82310
  * d/extra/dhclient-enter-resolved-hook:
    Check is-enabled systemd-resolved in dhclient hook (LP: #1853164)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=774c2f82a39a88fa0fd8b2adbfa0b8a8c3cd1fb5

 -- Dan Streetman <email address hidden> Thu, 26 Aug 2021 10:20:40 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.13

---------------
systemd (245.4-4ubuntu3.13) focal; urgency=medium

  * d/p/dell-clamshell-accel-location-base-with-sku.patch:
    Revert incorrect patch (LP: #1942899)

systemd (245.4-4ubuntu3.12) focal; urgency=medium

  [ Yao Wei ]
  * d/p/dell-clamshell-accel-location-base.patch:
    Add ACCEL_LOCATION=base property for Dell clamshell models (LP: #1938259)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=5c1be33900edee94da0dc9a4ade8edcd079b4c85

  [ Lukas Märdian ]
  * Add d/p/lp1934221-resolved-disable-event-sources-before-unreffing-them.patch
    - Fix segfault in systemd-resolve (LP: #1934221)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6c401900c70962052f56c7108fdc02fe7f84c9bf

  [ Simon Chopin ]
  * d/p/lp1914740-network-enable-DHCP-broadcast-flag-if-required-by-in.patch:
    - Apply upstream patch to fix Hipersocket DHCP mode (LP: #1914740)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=326ae43b7966d9e7c5f7124027185a79a07fa276

  [ Dan Streetman ]
  * d/p/lp1934981-correct-suspend-then-sleep-string.patch:
    Fix sleep verb used by logind during suspend-then-hibernate
    (LP: #1934981)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=997f3a7da3d5db22e3c63626c3f7dc3dff0830b0
  * d/p/lp1937238-util-return-the-correct-correct-wd-from-inotify-help.patch:
    Fix watch for time sync (LP: #1937238)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=dbabff8a03eb232c19174eff1335cd7cb7d7860c
  * d/extra/dhclient-enter-resolved-hook:
    Reset start limit counter for systemd-resolved in dhclient hook
    (LP: #1939255)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9d3a91a0b70a4b2bcc166f366cd0a880fd494812
  * d/p/lp1935051-shared-unit-file-make-sure-the-old-hashmaps-and-sets.patch:
    Fix memory leak in path cache (LP: #1935051)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=12d6bdeb35f309158fe8d4242c6dd9be4d067604
  * d/p/lp1934147/0001-cgroup-do-catchup-for-unit-cgroup-inotify-watch-file.patch,
    d/p/lp1934147/0002-core-Make-sure-cgroup_oom_queue-is-flushed-on-manage.patch:
    Catchup cgroup inotify watch after reexec/reload (LP: #1934147)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=63eabc88b8e0005eb40b15b543538ce35377bdbd

 -- Dan Streetman <email address hidden> Tue, 07 Sep 2021 14:37:22 -0400

[Dmitry Teselkin (teselkin-d)]

Hi. It seems this issue may occur not only when receiving addresses on multiple interfaces, but also when dhcp requests on multiple interfaces times out and the hook script was called with 'reason' == FAIL.
Found on Ubuntu 20.04.3 LTS, systemd is 245.4-4ubuntu3.15.

[Dan Streetman (ddstreet)]

> It seems this issue may occur not only when receiving addresses on multiple interfaces, but also when dhcp requests on multiple interfaces times out and the hook script was called with 'reason' == FAIL.

yep, the try-reload-or-restart is done for those as well and this should be fixed for those $reason cases; since this bug is already resolved, I suggest you open a new bug.