systemd-binfmt.service fails in tests-in-lxd

[Nick Rosbrook]

[Impact]

systemd-binfmt.service fails in LXD because access /proc/sys/binfmt_misc is prevented by LXD apparmor config. This causes our autopkgtest to fail, and leaves systemd-binfmt.service in a failed state:

root@jammy:~# systemctl status systemd-binfmt
× systemd-binfmt.service - Set Up Additional Binary Formats
     Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
     Active: failed (Result: exit-code) since Thu 2023-08-17 18:54:18 UTC; 1min 12s ago
       Docs: man:systemd-binfmt.service(8)
             man:binfmt.d(5)
             https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
   Main PID: 118 (code=exited, status=1/FAILURE)
        CPU: 5ms

Aug 17 18:54:18 jammy systemd[1]: Starting Set Up Additional Binary Formats...
Aug 17 18:54:18 jammy systemd-binfmt[118]: Failed to add binary format: Permission denied
Aug 17 18:54:18 jammy systemd[1]: systemd-binfmt.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 18:54:18 jammy systemd[1]: systemd-binfmt.service: Failed with result 'exit-code'.
Aug 17 18:54:18 jammy systemd[1]: Failed to start Set Up Additional Binary Formats.

[Test Plan]

1. test_no_failed from boot-and-services should pass.
2. Manual test:

* Start a Jammy LXD container:

$ lxc launch ubuntu-daily:jammy jammy
$ lxc exec jammy bash

* In the container, check systemd-binfmt.service:
root@jammy:~# systemctl status systemd-binfmt
● systemd-binfmt.service - Set Up Additional Binary Formats
     Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
    Drop-In: /etc/systemd/system/systemd-binfmt.service.d
             └─override.conf
     Active: active (exited) since Thu 2023-08-17 19:01:00 UTC; 2min 27s ago
       Docs: man:systemd-binfmt.service(8)
             man:binfmt.d(5)
             https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
   Main PID: 122 (code=exited, status=0/SUCCESS)
        CPU: 5ms

Aug 17 19:01:00 jammy systemd[1]: Starting Set Up Additional Binary Formats...
Aug 17 19:01:00 jammy systemd[1]: Finished Set Up Additional Binary Formats.
root@jammy:~# journalctl -t systemd-binfmt -b
Aug 17 19:01:00 jammy systemd-binfmt[122]: /proc/sys/fs/binfmt_misc is not mounted in read-write mode, skipping.

[ Where problems could occur]

The patches are contained to binfmt logic in systemd, and so any regressions would be within systemd-binfmt itself. The patches make it so that systemd-binfmt checks if /proc/sys/binfmt_misc is mounted as rw before trying to make any changes. If it is mounted, but read-only, then systemd-binfmt will exit cleanly and log a message indicating so. If the patch was incorrect, it could make it so that systemd-binfmt is not fully executed on systems where it should be.

[Original Description]

The systemd-binfmt.service requires read-write access to /proc/sys/fs/binfmt_misc, but this is not possible in unprivileged LXD containers without binfmt_misc namespace support in the kernel [1]. When this service is triggered, we get the following failure from test_no_failed within tests-in-lxd:

No failed units ... -------- journal for failed service systemd-binfmt.service -----------
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: Starting Set Up Additional Binary Formats...
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: Failed to flush binfmt_misc rules, ignoring: Permission denied
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: /usr/lib/binfmt.d/python3.10.conf:1: Failed to delete rule 'python3.10', ignoring: Permission denied
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd-binfmt[99]: /usr/lib/binfmt.d/python3.10.conf:1: Failed to add binary format 'python3.10': Permission denied
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: systemd-binfmt.service: Main process exited, code=exited, status=1/FAILURE
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: systemd-binfmt.service: Failed with result 'exit-code'.
Dec 08 10:43:37 autopkgtest-lxd-qicaxl systemd[1]: Failed to start Set Up Additional Binary Formats.
FAIL
test_rsyslog (__main__.ServicesTest) ... ok
test_tmp_cleanup (__main__.ServicesTest) ... ok
test_tmp_mount (__main__.ServicesTest) ... ok
test_udev (__main__.ServicesTest) ... skipped 'udev does not work in containers'

======================================================================
FAIL: test_no_failed (__main__.ServicesTest)
No failed units
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.METFVA/build.LjB/real-tree/debian/tests/boot-and-services", line 74, in test_no_failed
    self.assertEqual(failed, [])
AssertionError: Lists differ: ['systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats'] != []

First list contains 1 additional elements.
First extra element 0:
'systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats'

- ['systemd-binfmt.service loaded failed failed Set Up Additional Binary Formats']
+ []

The systemd-binfmt.service is only started if there is a configuration to be loaded from one of the valid configuration directories (e.g. /usr/lib/binfmt.d/python3.10.conf). Previously, no such file was installed in the test LXD images, but a recent upload of python3.10 added support for systemd-binfmt [2].

[1] https://discuss.linuxcontainers.org/t/3-1-0-binfmt-support-service-in-unprivileged-guest-requires-write-access-on-hosts-proc-sys-fs-binfmt-misc/5249
[2] https://launchpadlibrarian.net/632281245/python3.10_3.10.8-1_3.10.8-3.diff.gz

[Nick Rosbrook (enr0n)]

This has been resolved upstream[1] and I will bring the patches into lunar.

[1] https://github.com/systemd/systemd/pull/25693

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 252.4-1ubuntu1

---------------
systemd (252.4-1ubuntu1) lunar; urgency=medium

  * Drop oomd-fix-unreachable-test-case-in-test-oomd-util.patch.
    Applied upstream: https://github.com/systemd/systemd-stable/commit/bcd42b3c88
    File: debian/patches/oomd-fix-unreachable-test-case-in-test-oomd-util.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=93cf2f4709a6768d6eeb04153c92bffec0726753
  * Drop oomd-always-allow-root-owned-cgroups-to-set-ManagedOOMPre.patch.
    Applied upstream: https://github.com/systemd/systemd-stable/commit/2bdf5b0382
    File: debian/patches/oomd-always-allow-root-owned-cgroups-to-set-ManagedOOMPre.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=24f51625bcec9a7b4f32a0caf363b000b173748b
  * debian/rules: remove explicit -fcf-protection flag.
    This is included in the default gcc flags on Ubuntu[1], so we do not need
    this anymore. Drop the diff to reduce our delta from Debian.
    [1] https://wiki.ubuntu.com/ToolChain/CompilerFlags#A-fcf-protection
    File: debian/rules
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b72bf5bb964aa89769417491fcc99926a06490ba
  * Revert "Drop systemd.prerm safety check."
    This reverts commit 0244c4d56556317f14eecc2f51871969ef02ba7b.
    systemd does not run in a chroot, so /run/systemd/system should not
    exist in chroots. Debian has kept this without issue, so revert this in
    order to reduce delta with Debian.
    File: debian/systemd.prerm
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d7df55f11603022cde6cf897aaddc02624e79336
  * Drop 0001-Revert-tests-add-test-case-for-UMask-BindPaths-combi.patch.
    This patch is specific to privileged LXD containers, which we are not
    testing anymore, so there is no need to carry the patch anymore.
    File: debian/patches/0001-Revert-tests-add-test-case-for-UMask-BindPaths-combi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2acbc4741346c684ea3fa8aecf1d6ed88a7444cd
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275-binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275-binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275-binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275-stat-util-introduce-fd_is_read_only_fs.patch
    - debian/patches/lp1999275-unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=41fa3a0420c5633200ecd60add6cadb149c98ccd
  * debian/patches: update test-skip-some-tests-when-machine-id-is-not-initialized.patch.
    The latest stable tag includes some, but not all patches which [1] needs
    to work correctly. For now, just simplify the `machine_id_initialized()`
    helper and make a note in the patch that the changes were forwarded
    upstream.
    [1] https://github.com/systemd/systemd/pull/25732
    File: debian/patches/test-skip-some-tests-whe...

[Nick Rosbrook (enr0n)]

This affects Jammy now too due to python3.10 SRU.

[Łukasz Zemczak (sil2100)]

Thank you for the manual tests here. Yes, as this changes core code, we need manual testing besides the autopkgtest one.

[Łukasz Zemczak (sil2100)]

Hello Nick, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.10) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.4.10 (armhf)
casync/2+20201210-1build1 (ppc64el)
comitup/1.15-1 (armhf)
dbus/1.12.20-2ubuntu4.1 (armhf)
initramfs-tools/0.140ubuntu13.4 (s390x)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-gcp-6.2/6.2.0-1011.11~22.04.3 (arm64)
linux-lowlatency/5.15.0-83.92 (arm64)
linux-lowlatency-hwe-5.19/5.19.0-1030.30 (arm64)
linux-nvidia-tegra/5.15.0-1016.16 (arm64)
linux-oracle-5.19/5.19.0-1027.30 (arm64)
mkosi/unknown (s390x)
munin/2.0.57-1ubuntu2 (armhf)
netplan.io/0.105-0ubuntu2~22.04.3 (arm64)
prometheus-postfix-exporter/unknown (s390x)
samba/2:4.15.13+dfsg-0ubuntu1.4 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Nick Rosbrook (enr0n)]

I have verified the fix using systemd 249.11-0ubuntu3.10 from jammy-proposed:

1. Verified that boot-and-services is passing on all arches for 249.11-0ubuntu3.10: https://autopkgtest.ubuntu.com/packages/systemd.

2. Manual test:

root@jammy:~# systemd-detect-virt
lxc
root@jammy:~# apt-cache policy systemd
systemd:
  Installed: 249.11-0ubuntu3.10
  Candidate: 249.11-0ubuntu3.10
  Version table:
 *** 249.11-0ubuntu3.10 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     249.11-0ubuntu3.9 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     249.11-0ubuntu3.7 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     249.11-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages
root@jammy:~# systemctl status systemd-binfmt.service
● systemd-binfmt.service - Set Up Additional Binary Formats
     Loaded: loaded (/lib/systemd/system/systemd-binfmt.service; static)
    Drop-In: /etc/systemd/system/systemd-binfmt.service.d
             └─override.conf
     Active: active (exited) since Mon 2023-08-28 19:22:15 UTC; 13s ago
       Docs: man:systemd-binfmt.service(8)
             man:binfmt.d(5)
             https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 115 ExecStart=/lib/systemd/systemd-binfmt (code=exited, status=0/SUCCESS)
   Main PID: 115 (code=exited, status=0/SUCCESS)
        CPU: 4ms

Aug 28 19:22:15 jammy systemd[1]: Starting Set Up Additional Binary Formats...
Aug 28 19:22:15 jammy systemd[1]: Finished Set Up Additional Binary Formats.
root@jammy:~# journalctl -t systemd-binfmt -b
Aug 28 19:22:15 jammy systemd-binfmt[115]: /proc/sys/fs/binfmt_misc is not mounted in read-write mode, skipping.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

The release of this SRU has been rolled back in jammy because of bug #2035406 which appears to be a regression introduced in the new version of systemd.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...