systemd-resolved, systemd-networkd and others fail to start in lxc container with v240 systemd

[Robie Basak]

This is a regression from 239-7ubuntu15 to 240-5ubuntu1.

Steps to reproduce:

lxc launch ubuntu-daily:disco rbasak-resolv
lxc exec rbasak-resolv bash
systemctl status systemd-resolved # observe running
echo "deb http://archive.ubuntu.com/ubuntu/ disco-proposed main universe multiverse restricted" >> /etc/apt/sources.list
apt update
# Update to 240-5ubuntu1 from proposed
apt install systemd libsystemd0 systemd-sysv libnss-systemd libpam-systemd
reboot
lxc exec rbasak-resolv bash
systemctl status systemd-resolved # observe failed

● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-01-28 16:50:37 UTC; 2min 28s ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
  Process: 290 ExecStart=/lib/systemd/systemd-resolved (code=exited, status=226/NAMESPACE)
 Main PID: 290 (code=exited, status=226/NAMESPACE)

Jan 28 16:50:37 rbasak-resolv systemd[1]: systemd-resolved.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Jan 28 16:50:37 rbasak-resolv systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5.
Jan 28 16:50:37 rbasak-resolv systemd[1]: Stopped Network Name Resolution.
Jan 28 16:50:37 rbasak-resolv systemd[1]: systemd-resolved.service: Start request repeated too quickly.
Jan 28 16:50:37 rbasak-resolv systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Jan 28 16:50:37 rbasak-resolv systemd[1]: Failed to start Network Name Resolution.

This causes /etc/resolv.conf to point to a file that isn't created, so all name resolution fails. As far as I can determine, landing this in the release pocket would cause all default LXD containers to stop working.

In my case it breaks "autopkgtest -U --apt-pocket=proposed ... -- lxd ubuntu-daily:disco"

Tagging block-proposed as migration would regress the release pocket, and marking Critical as it breaks the system (presumably only in a container though, and it is only in proposed currently).

=== Workaround ===

$ lxc config set test-v240 raw.apparmor 'mount options=(ro,nodev,remount,bind),
mount options=(ro,nosuid,nodev,remount,bind),
mount options=(ro,nosuid,noexec,remount,strictatime),
mount options=(ro,nosuid,noexec,remount,bind,strictatime),
mount options=(ro,nosuid,nodev,noexec,remount,bind),'

[Robie Basak (racb)]

If it matters, my host is running Disco, upgraded nightly but last rebooted 59 days ago. lxd is from the snap:

installed: 3.9 (9919) 54MB -

[Thomas Ward (teward)]

Robie:

Confirmed with a Bionic host as well using the same LXD snap and environment, and the same reproduction steps.

(Marking as "Confirmed")

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Jan 28 23:50:06 ottawa audit[10278]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-improved-kodiak_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/home/" pid=10278 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
Jan 28 23:50:06 ottawa kernel: audit: type=1400 audit(1548719406.237:332): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-improved-kodiak_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/home/" pid=10278 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
Jan 28 23:50:06 ottawa audit[10310]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-improved-kodiak_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/home/" pid=10310 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
Jan 28 23:50:06 ottawa kernel: audit: type=1400 audit(1548719406.273:333): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-improved-kodiak_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/home/" pid=10310 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"

So systemd v240 tries to setup mount namespace to further contain execution, and it appears that this is no longer possible inside the lxd container, due to apparmor denies.

I'm not sure if this is a bug/feature of systemd | snapd | lxd | apparmor, as all of these are involved.

[Dimitri John Ledkov (xnox)]

Hmmm, but it should be fine if one gets EPERM or like EACCES. Hmmm..

[Dimitri John Ledkov (xnox)]

root@improved-kodiak:~# systemctl restart systemd-resolved
Jan 29 00:11:52 improved-kodiak systemd[1]: Bus private-bus-connection: changing state UNSET → OPENING
Jan 29 00:11:52 improved-kodiak systemd[1]: Bus private-bus-connection: changing state OPENING → AUTHENTICATING
Jan 29 00:11:52 improved-kodiak systemd[1]: Accepted new private connection.
Jan 29 00:11:52 improved-kodiak systemd[1]: Bus private-bus-connection: changing state AUTHENTICATING → RUNNING
Jan 29 00:11:52 improved-kodiak systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=RestartUnit cookie=1 reply_cookie=0 signature=ss error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: systemd-resolved.service: Trying to enqueue job systemd-resolved.service/restart/replace
Jan 29 00:11:52 improved-kodiak systemd[1]: systemd-resolved.service: Installed new job systemd-resolved.service/restart as 1358
Jan 29 00:11:52 improved-kodiak systemd[1]: systemd-resolved.service: Enqueued job systemd-resolved.service/restart as 1358
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dresolved_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=1 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dresolved_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=2 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dresolved_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=6343 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1/unit/systemd_2dresolved_2eservice interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=6344 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=org.freedesktop.systemd1 destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew cookie=3 reply_cookie=0 signature=uos error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew cookie=6345 reply_cookie=0 signature=uos error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=4 reply_cookie=1 signature=o error-name=n/a error-message=n/a
Jan 29 00:11:52 improved-kodiak systemd[1]...

[Dimitri John Ledkov (xnox)]

This is similar to https://github.com/systemd/systemd/issues/10032
But this time around with apparmor, under lxd, on ubuntu
Regression from 1beab8b0d0ff2d7d1436b52d4a0c3d56dc908962
Will be tracing this further now.

[Dimitri John Ledkov (xnox)]

Actually more is needed:

$ lxc config set test-v240 raw.apparmor 'mount options=(ro,nodev,remount,bind),
mount options=(ro,nosuid,nodev,remount,bind),
mount options=(ro,nosuid,noexec,remount,strictatime),
mount options=(ro,nosuid,noexec,remount,bind,strictatime),
mount options=(ro,nosuid,nodev,noexec,remount,bind),'

[Dimitri John Ledkov (xnox)]

Removing block-proposed, although lxd fix is in flight, I have reverted the relevant commit for now in the systemd in disco-proposed.

The intention is to still ship systemd in disco with that patch in place, and thus fixed lxd is still needed in v3.9+ and v3.0 series.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 240-5ubuntu3

---------------
systemd (240-5ubuntu3) disco; urgency=medium

  * debian/tests: blacklist upstream test-24-unit-tests on ppc64le.
    Fails, not a regression as it's a new test case, which was never before
    executed on ppc64le.
    File: debian/tests/upstream
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8062b9a2712c390010d2948eaf764a1b52e68715

 -- Dimitri John Ledkov <email address hidden> Sat, 02 Feb 2019 11:05:12 +0100