systemd-resolved unit should run Before=network-online.target

[Ryan Harper]

=== Begin SRU Template ===
[Impact]

For releases using systemd-resolved (yakkety and zesty); the unit
configuration does not require that the service be active before
allowing systemd to reach 'network-online.target' which is a special
target used to allow other units which require networking access to
run.

In some cases, units which run After=network-online.target may
encounter DNS failures if systemd-resolved is not yet completely
active.

The fix is to add Before=network-online.target to the Unit directives
for systemd-resolved.service.

[Test Case]

1. lxc launch ubuntu-daily:yakkety y1
2. lxc exec y1 -- journalctl -o short-precise \
   --unit systemd-resolved --unit network-online.target

3. Check order of units; If 'Reached target Network is Online' is
   listed before 'Started Network Name Resolution', then DNS may not
   be up.

Example FAIL output:

# apt-cache policy systemd
systemd:
  Installed: 231-9ubuntu3
  Candidate: 231-9ubuntu3
  Version table:
 *** 231-9ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu yakkety-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     231-9git1 500
        500 http://archive.ubuntu.com/ubuntu yakkety/main amd64 Packages

# journalctl -o short-precise -u systemd-resolved -u network-online.target
-- Logs begin at Thu 2017-03-23 21:22:42 UTC, end at Thu 2017-03-23 21:22:49 UTC. --
Mar 23 21:22:47.173454 y1 systemd[1]: Reached target Network is Online.
Mar 23 21:22:47.197566 y1 systemd[1]: systemd-resolved.service: Failed to reset devices.list: Operation not permitted
Mar 23 21:22:47.198023 y1 systemd[1]: Starting Network Name Resolution...
Mar 23 21:22:47.207216 y1 systemd-resolved[438]: Positive Trust Anchors:
Mar 23 21:22:47.207265 y1 systemd-resolved[438]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde3
Mar 23 21:22:47.207319 y1 systemd-resolved[438]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-add
Mar 23 21:22:47.216370 y1 systemd-resolved[438]: Using system hostname 'y1'.
Mar 23 21:22:47.237441 y1 systemd-resolved[438]: Switching to system DNS server 10.245.119.1.
Mar 23 21:22:47.399614 y1 systemd[1]: Started Network Name Resolution.

Example PASS output:
# journalctl -o short-precise -u systemd-resolved -u network-online.target
-- Logs begin at Thu 2017-03-23 21:25:08 UTC, end at Thu 2017-03-23 21:25:11 UTC. --
Mar 23 21:25:10.206276 y1 systemd[1]: systemd-resolved.service: Failed to reset devices.list: Operation not permitted
Mar 23 21:25:10.206685 y1 systemd[1]: Starting Network Name Resolution...
Mar 23 21:25:10.229430 y1 systemd-resolved[445]: Positive Trust Anchors:
Mar 23 21:25:10.229449 y1 systemd-resolved[445]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde3
Mar 23 21:25:10.229491 y1 systemd-resolved[445]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-add
Mar 23 21:25:10.229759 y1 systemd-resolved[445]: Using system hostname 'y1'.
Mar 23 21:25:10.231969 y1 systemd-resolved[445]: Switching to system DNS server 10.245.119.1.
Mar 23 21:25:10.291591 y1 systemd[1]: Started Network Name Resolution.
Mar 23 21:25:10.291944 y1 systemd[1]: Reached target Network is Online.

[Regression Potential]
Changing order in boot can be dangerous. This is a possiblity of
units using the defaults in /etc/resolv.conf (which doesn't point to
systemd-resolved until later during boot) would now run when
/etc/resolv.conf points to systemd-resolved service (127.0.0.53).

[Original Description]
1) Xenial, Yakkety and Zesty; (Xenial is affected if you're using networkd and resolved, but it's not the default)
2) 229-4ubuntu16, 231-9ubuntu3, 232-18ubuntu1 respectively to (1)

3) DNS resolution should be available once systemd has reached 'network-online.target' state
4) Sometimes systemd-resolved has not become active prior to network-online.target and DNS service is not available.

The remaining issue for the systemd-resolved.service unit is that it needs to include a Before=network-online.target to ensure it's ordered to run before systemd reaches 'network-online.target'

[Steve Langasek (vorlon)]

Related history in LP: #1649931

[Brian Murray (brian-murray)]

Missing SRU information regarding Impact, Test Case, Regression Potential.

[Ryan Harper (raharper)]

Working it now, sorry for the delay

On Thu, Mar 23, 2017 at 1:07 PM, Brian Murray <email address hidden> wrote:

> Missing SRU information regarding Impact, Test Case, Regression
> Potential.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1673860
>
> Title:
> systemd-resolved unit should run Before=network-online.target
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/
> 1673860/+subscriptions
>

[Łukasz Zemczak (sil2100)]

The zesty systemd upload with this fix is still in zesty-proposed, please make sure to carry it through. The yakkety systemd SRU that includes this fix as part of its changes cannot be approved before 232-21ubuntu1 migrates to the release pocket.

[Ryan Harper (raharper)]

I just tested the zesty-proposed version and it works as expected.

% lxc launch ubuntu-daily:zesty z1
Creating z1
Starting z1

# confirm current version of systemd
% lxc exec z1 -- apt-cache policy systemd
systemd:
  Installed: 232-19
  Candidate: 232-19
  Version table:
 *** 232-19 500
        500 http://archive.ubuntu.com/ubuntu zesty/main amd64 Packages
        100 /var/lib/dpkg/status

# confirm that network-online.target is started before systemd-resolved
%3 lxc exec z1 -- journalctl --no-pager -o short-precise --unit systemd-resolved --unit network-online.target
-- Logs begin at Fri 2017-03-31 18:48:46 UTC, end at Fri 2017-03-31 18:48:54 UTC. --
Mar 31 18:48:51.485348 z1 systemd[1]: Reached target Network is Online.
Mar 31 18:48:51.519147 z1 systemd[1]: systemd-resolved.service: Failed to reset devices.list: Operation not permitted
Mar 31 18:48:51.526564 z1 systemd[1]: systemd-resolved.service: Failed to set invocation ID on control group /system.slice/systemd-resolved.service, ignoring: Operation not permitted
Mar 31 18:48:51.533523 z1 systemd[1]: Starting Network Name Resolution...
Mar 31 18:48:51.665835 z1 systemd-resolved[432]: Positive Trust Anchors:
Mar 31 18:48:51.670257 z1 systemd-resolved[432]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Mar 31 18:48:51.671291 z1 systemd-resolved[432]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Mar 31 18:48:51.672004 z1 systemd-resolved[432]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Mar 31 18:48:51.673770 z1 systemd-resolved[432]: Using system hostname 'z1'.
Mar 31 18:48:51.696484 z1 systemd-resolved[432]: Switching to system DNS server 10.245.119.1.
Mar 31 18:48:51.807829 z1 systemd[1]: Started Network Name Resolution.

# enable proposed
% lxc exec z1 -- sh -c 'echo "deb http://us.archive.ubuntu.com/ubuntu/ zesty-proposed main restricted" > /etc/apt/sources.list.d/zesty-proposed.list; apt update'
Hit:1 http://security.ubuntu.com/ubuntu zesty-security InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu zesty-proposed InRelease [240 kB]
Get:3 http://archive.ubuntu.com/ubuntu zesty InRelease [243 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu zesty-proposed/main amd64 Packages [53.3 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu zesty-proposed/main Translation-en [26.3 kB]
Hit:6 http://archive.ubuntu.com/ubuntu zesty-updates InRelease
Get:7 http://archive.ubuntu.com/ubuntu zesty-backports InRelease [92.0 kB]
Get:8 http://archive.ubuntu.com/ubuntu zesty/main Sources [911 kB]
Get:9 http://archive.ubuntu.com/ubuntu zesty/multiverse Sources [188 kB]
Get:10 http://archive.ubuntu.com/ubuntu zesty/universe Sources [8481 kB]
Get:11 http://archive.ubuntu.com/ubuntu zesty/restricted Sources [507...

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 232-21ubuntu2

---------------
systemd (232-21ubuntu2) zesty; urgency=medium

  * pkgconfig: Cherrypick upstream fix to libdir locations in .pc files
    (LP: #1674201)

 -- Dimitri John Ledkov <email address hidden> Tue, 28 Mar 2017 16:59:14 +0100

[Łukasz Zemczak (sil2100)]

Hello Ryan, or anyone else affected,

Accepted systemd into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/231-9ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

Confirmed this fix in yakkety. Behavior with systemd 231-9ubuntu3:

# journalctl -o short-precise -u systemd-resolved -u network-online.target --no-pager
-- Logs begin at Thu 2017-04-20 23:50:20 UTC, end at Fri 2017-04-21 03:22:45 UTC
Apr 20 23:51:06.897218 ubuntu systemd[1]: Starting Network Name Resolution...
Apr 20 23:51:06.899220 ubuntu systemd[1]: Reached target Network is Online.
Apr 20 23:51:12.471807 ubuntu systemd-resolved[1116]: Positive Trust Anchors:
[...]
Apr 20 23:51:19.152992 ubuntu systemd[1]: Started Network Name Resolution.

Behavior with systemd 231-9ubuntu4 (with an extra ExecStartPre=/bin/sleep 10 in systemd-resolved.service for good measure):

# journalctl -o short-precise -u systemd-resolved -u network-online.target --no-pager
-- Logs begin at Fri 2017-04-21 03:27:20 UTC, end at Fri 2017-04-21 03:27:47 UTC. --
Apr 21 03:27:26.077418 ubuntu systemd[1]: Starting Network Name Resolution...
Apr 21 03:27:36.134810 ubuntu systemd-resolved[1119]: Positive Trust Anchors:
[...]
Apr 21 03:27:36.174373 ubuntu systemd[1]: Started Network Name Resolution.
Apr 21 03:27:38.955658 ubuntu systemd[1]: Reached target Network is Online.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 231-9ubuntu4

---------------
systemd (231-9ubuntu4) yakkety; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: if
    resolved is going to be started, make sure this blocks
    network-online.target. LP: #1673860.
  * debian/patches/resolved-follow-CNAMES-for-DNS-stub-replies.patch:
    Cherry-pick upstream fix for resolved failing to follow CNAMES for DNS
    stub replies. LP: #1647031.
  * debian/patches/logind-update-empty-and-infinity-handling-for-User-T.patch:
    Cherry-pick upstream fix to handle empty and "infinity" values for
    [User]TasksMax. Closes LP: #1651518.

 -- Steve Langasek <email address hidden> Mon, 20 Mar 2017 22:14:14 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Ryan, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

Since resolved is not enabled by default in xenial, I'm doing a slightly modified testcase of enabling resolved first, reboot, upgrade to new systemd, reboot again, check the results after every reboot.

Package versions from the upgrade log: Unpacking systemd (229-4ubuntu18) over (229-4ubuntu17)

Reached target Network is Online has become the last entry in the journal _after_ the Starting Network Name Resolution. Previously it was before.

[Adam Conrad (adconrad)]

Hello Ryan, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dimitri John Ledkov (xnox)]

Starting with systemd 229-4ubuntu17.
Observed:
Jul 19 13:26:15 key-giraffe systemd[1]: Reached target Network is Online.
Jul 19 13:26:15 key-giraffe systemd[1]: Starting Network Name Resolution...

Upgraded to systemd 229-4ubuntu19

Observed:
Jul 19 13:32:58 key-giraffe systemd[1]: Started Network Name Resolution.
Jul 19 13:32:58 key-giraffe systemd[1]: Reached target Network is Online

All is good.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu19

---------------
systemd (229-4ubuntu19) xenial; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: partially
    revert, by removing ExecStart|StopPost lines, as these are not needed on
    xenial and generate warnings in the journal. (LP: #1704677)

systemd (229-4ubuntu18) xenial; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: if resolved
    is going to be started, make sure this blocks network-online.target.
    (LP: #1673860)
  * networkd: cherry-pick support for setting bridge port's priority
    (LP: #1668347)
  * Cherrypick upstream commit to enable system use kernel maximum limit for
    RLIMIT_NOFILE isntead of hard-coded (low) limit of 65536. (LP: #1686361)
  * Cherrypick upstream patch for platform predictable interface names.
    (LP: #1686784)
  * resolved: fix null pointer dereference crash (LP: #1621396)
  * Cherrypick core/timer downgrade message about random time addition
    (LP: #1692136)
  * SECURITY UPDATE: Out-of-bounds write in systemd-resolved (LP: #1695546)
    - CVE-2017-9445
  * Cherry-pick subset of patches to introduce infinity value in logind.conf
    for UserTasksMax (LP: #1651518)

 -- Dimitri John Ledkov <email address hidden> Mon, 17 Jul 2017 17:00:42 +0100