systemd: autopkgtest: tests-in-lxd fails because of remaining snap .mount units

[Nick Rosbrook]

[Impact]

 * The "tests-in-lxd/boot-and-services/test_no_failed" autopkgtest fails
 * it did not stand out due to LP: #1976607
 * we want to have proper autopkgtest coverage for systemd in Jammy LTS

[Test Plan]

 * check autopkgtest logs on autopkgtest.ubuntu.com and make sure all instances of "boot-and-services" PASS (especially the "test_no_failed" test case. This includes running the test natively in autopkgtest and inside the "tests-in-lxd" container.

[Where problems could occur]

 * other/new snapd-*.mount regressions could be hidden.
 * but snapd is being removed during systemd's tests anyway (introducing this failure)

[Other Info]

 * The issue was not visible up to now, as the whole tests-in-lxd autopkgtests has been skipped: LP: #1976607

=== original description ===
This build log provides an example of the problem: https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/ppc64el/s/systemd/20220331_190433_fdbbe@/log.gz.

During setup for tests-in-lxd, an attempt is made to purge snapd, but this fails:

[ ... ]
Purging configuration files for snapd (2.54.3+21.10.1ubuntu0.2) ...
Stopping snap.lxd.activate.service
Stopping unit snap.lxd.activate.service
Waiting until unit snap.lxd.activate.service is stopped [attempt 1]
snap.lxd.activate.service is stopped.
Removing snap.lxd.activate.service
Stopping snap.lxd.daemon.service
Stopping unit snap.lxd.daemon.service
Waiting until unit snap.lxd.daemon.service is stopped [attempt 1]
snap.lxd.daemon.service is stopped.
Removing snap.lxd.daemon.service
Stopping snap.lxd.user-daemon.service
Stopping unit snap.lxd.user-daemon.service
Waiting until unit snap.lxd.user-daemon.service is stopped [attempt 1]
snap.lxd.user-daemon.service is stopped.
Removing snap.lxd.user-daemon.service
Stopping snap-core20-1378.mount
Stopping unit snap-core20-1378.mount
Waiting until unit snap-core20-1378.mount is stopped [attempt 1]
snap-core20-1378.mount is stopped.
Removing snap core20 and revision 1378
Removing snap-core20-1378.mount
Stopping snap-lxd-22672.mount
Stopping unit snap-lxd-22672.mount
Waiting until unit snap-lxd-22672.mount is stopped [attempt 1]
snap-lxd-22672.mount is stopped.
Removing snap lxd and revision 22672
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/meminfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/stat': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/uptime': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/diskstats': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/swaps': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/loadavg': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/slabinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpuidle': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/uevent': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/svm': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/hotplug': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/offline': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/present': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/power': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/dscr_default': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/probe': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/smt': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/vulnerabilities': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpufreq': Operation not permitted
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/isolated': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/kernel_max': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/modalias': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/cgroup': Input/output error
dpkg: error processing package snapd (--purge):
 installed snapd package post-removal script subprocess returned error exit status 1
[ ... ]

This appears to result in some snap .mount units being left over in /etc/systemd/system. These leftover mount units will now always fail because snapd etc. is no longer installed. Thus, the 'test_no_failed' test, which checks for any failed units after boot, will fail:

[ ... ]
======================================================================
FAIL: test_no_failed (__main__.ServicesTest)
No failed units
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.s5bwuE/build.2Vb/real-tree/debian/tests/boot-and-services", line 69, in test_no_failed
    self.assertEqual(failed, [])
AssertionError: Lists differ: ['snap-lxd-22672.mount loaded failed fai[119 chars]180'] != []

First list contains 2 additional elements.
First extra element 0:
'snap-lxd-22672.mount loaded failed failed Mount unit for lxd, revision 22672'

+ []
- ['snap-lxd-22672.mount loaded failed failed Mount unit for lxd, revision '
- '22672',
- 'snap-snapd-15180.mount loaded failed failed Mount unit for snapd, revision '
- '15180']

----------------------------------------------------------------------
Ran 23 tests in 3.771s

FAILED (failures=1, skipped=7)
autopkgtest [18:16:43]: test boot-and-services: -----------------------]
autopkgtest [18:16:44]: test boot-and-services: - - - - - - - - - - results - - - - - - - - - -
boot-and-services FAIL non-zero exit status 1
autopkgtest [18:16:44]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
autopkgtest [18:16:52]: test tests-in-lxd: -----------------------]
autopkgtest [18:16:54]: test tests-in-lxd: - - - - - - - - - - results - - - - - - - - - -
tests-in-lxd FAIL non-zero exit status 1
[ ... ]

The underlying snapd bug appears to be related to bug 1903967 and bug 1950864. In the mean time, the test should be fixed so this does not block systemd autopktests.

[Brian Murray (brian-murray)]

Hello Nick, or anyone else affected,

Accepted systemd into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/248.3-1ubuntu8.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (248.3-1ubuntu8.5) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

snapd/2.54.3+21.10.1ubuntu0.2 (s390x, ppc64el, arm64, amd64)
casper/1.465 (amd64)
tpm2-abrmd/unknown (ppc64el)
corosync-qdevice/unknown (ppc64el)
nftables/unknown (ppc64el)
udisks2/2.9.4-1 (arm64)
diaspora-installer/0.7.15.0+debian1 (arm64, s390x)
systemd/248.3-1ubuntu8.5 (armhf)
swupdate/2020.11-2 (s390x)
munin/2.0.57-1ubuntu1 (arm64, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Nick Rosbrook (enr0n)]

The autopkgtest regressions blocking systemd 248.3-1ubuntu8.5 in impish-proposed have been resolved. The regressions were caused either by (1) network/infrastructure issues and succeeded on retry, or by (2) unrelated snapd regressions.

This change is verified by the fact that the systemd tests-in-lxd autopkgtest was fixed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 248.3-1ubuntu8.5

---------------
systemd (248.3-1ubuntu8.5) impish; urgency=medium

  * debian/tests/boot-and-services: Ignore failed snap mount units in test_no_failed (LP: #1967576)

systemd (248.3-1ubuntu8.4) impish; urgency=medium

  [ Lukas Märdian ]
  * Fix deadlock between pid1 and dbus-daemon (LP: #1871538)
    Author: Lukas Märdian
    File: debian/patches/pid1-set-SYSTEMD_NSS_DYNAMIC_BYPASS-1-env-var-for-dbus-da.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f79535077473902bad911dc2652a2fff4066fa30
  * Don't override Ubuntu's default sysctl values (LP: #1962038)
    Author: Lukas Märdian
    File: debian/patches/debian/UBUNTU-Don-t-override-Ubuntu-s-default-sysctl-values-LP-1962038.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3ba2764d8f77e616461c9722923f685fad79f8c6

systemd (248.3-1ubuntu8.3) impish; urgency=medium

  [ Jeremy Szu ]
  * Add a allowlist to unblock intel-hid on new HP machines (LP: #1955997)
    Author: Jeremy Szu
    File: debian/patches/lp1955997-add-a-allowlist-to-unblock-intel-hid-on-HP-mach.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=554d46e6a6ab80befd424ead7ffa8e6f993b5f66

 -- Nick Rosbrook <email address hidden> Fri, 01 Apr 2022 16:39:25 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "systemd-lp-1967576.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Lukas Märdian (slyon)]

Marking as "In Progress" (for Kinetic) according to: https://discourse.ubuntu.com/t/foundation-team-updates-thursday-09-june-2022/28788/9

[Steve Langasek (vorlon)]

Hello Nick, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lukas Märdian (slyon)]

I've tested systemd 249.11-0ubuntu3.3 from jammy-proposed, all tests passed, especially the "boot-and-services" test cases:

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/s/systemd/20220611_235311_cd56e@/log.gz
autopkgtest [23:52:23]: @@@@@@@@@@@@@@@@@@@@ summary
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
unit-config PASS
storage PASS
networkd-test.py PASS
build-login PASS
boot-and-services PASS
udev PASS
unit-tests PASS
tests-in-lxd PASS
upstream-1 PASS
upstream-2 PASS
boot-smoke PASS
systemd-fsckd SKIP exit status 77 and marked as skippable

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/s/systemd/20220611_225942_ad450@/log.gz
autopkgtest [22:59:15]: @@@@@@@@@@@@@@@@@@@@ summary
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
unit-config PASS
storage PASS
networkd-test.py PASS
build-login PASS
boot-and-services PASS
udev PASS
unit-tests PASS
tests-in-lxd PASS
upstream-1 PASS
upstream-2 PASS
boot-smoke PASS
systemd-fsckd SKIP exit status 77 and marked as skippable

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/armhf/s/systemd/20220611_151928_d218a@/log.gz
autopkgtest [15:19:21]: @@@@@@@@@@@@@@@@@@@@ summary
storage SKIP Test requires machine-level isolation but testbed does not provide that
udev SKIP Test requires machine-level isolation but testbed does not provide that
tests-in-lxd SKIP Test requires machine-level isolation but testbed does not provide that
upstream-1 SKIP Test requires machine-level isolation but testbed does not provide that
upstream-2 SKIP Test requires machine-level isolation but testbed does not provide that
systemd-fsckd SKIP Test requires machine-level isolation but testbed does not provide that
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
unit-config PASS
networkd-test.py PASS
build-login PASS
boot-and-services PASS
unit-tests PASS
boot-smoke PASS

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/ppc64el/s/systemd/20220613_115627_57124@/log.gz
autopkgtest [11:56:01]: @@@@@@@@@@@@@@@@@@@@ summary
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
unit-config PASS
storage PASS
networkd-test.py PASS
build-login PASS
boot-and-services PASS
udev PASS
unit-tests PASS
tests-in-lxd PASS
upstream-1 PASS
upstream-2 PASS
boot-smoke PASS
systemd-fsckd SKIP exit status 77 and marked as skippable

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/s390x/s/systemd/20220611_171627_8ac5...

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.3

---------------
systemd (249.11-0ubuntu3.3) jammy; urgency=medium

  [ Lukas Märdian ]
  * Build with and suggest fido2 and tpm libraries (LP: #1969375)
    These are used via dlopen only if available by some tools like
    systemd-cryptsetup, systemd-cryptenroll and systemd-repart,
    with graceful fallbacks if they are not found.
    Build-depend on them so that the features get compiled in
    (apart from stage1 builds), and add appropriate Suggests.
    Backport of:
    https://salsa.debian.org/systemd-team/systemd/-/commit/6b5e99f1d7f63c0c83007de9f98f7745f4a564f8
    Files:
    - debian/control
    - debian/rules
    - debian/tests/control
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c3f5affb669794f9ebfea8d81c68b1aacdde0511
  * Run tests-in-lxd autopkgtest via LXD snap, deb is no more (LP: #1976607)
    Files:
    - debian/tests/control
    - debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=eccfd52b275d1b0544dd44f858bcee8508c0957f

  [ Nick Rosbrook ]
  * d/t/boot-and-services: Ignore failed snap mount units in test_no_failed
    (LP: #1967576)
    Author: Nick Rosbrook
    File: debian/tests/boot-and-services
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf823bffe5cb47a6eb531d9869f69a844f356376

systemd (249.11-0ubuntu3.2) jammy; urgency=medium

  * d/p/lp1964494-network-do-not-enable-IPv4-ACD-for-IPv4-link-local-a.patch:
    do not enable IPv4 ACD for IPv4 link-local address if ACD is
    disabled explicitly (LP: #1964494)

 -- Lukas Märdian <email address hidden> Tue, 07 Jun 2022 12:49:20 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 251.2-2ubuntu1

---------------
systemd (251.2-2ubuntu1) kinetic; urgency=medium

  [ Nick Rosbrook ]
  * Merge to Ubuntu from Debian unstable
    - Dropped patches (applied upstream)
      + test-explicitly-configure-oomd-stuff-via-dropins.patch
      + test-enable-systemd-oomd.service.patch
      + linux-5.15-compat-ioprio/shared-split-out-ioprio-related-stuff-into-ioprio-util.-c.patch
      + linux-5.15-compat-ioprio/variuos-add-missing-includes.patch
      + linux-5.15-compat-ioprio/man-don-t-mention-IOSchedulingClass-none-anymore-in-the-d.patch
      + linux-5.15-compat-ioprio/test-add-test-for-ioprio-normalization.patch
      + linux-5.15-compat-ioprio/Define-ioprio_-get-set-the-same-as-other-compat-syscalls.patch
      + linux-5.15-compat-ioprio/Get-rid-of-ioprio.h-and-add-a-minimalistic-reimplementati.patch
      + linux-5.15-compat-ioprio/ioprio-util-add-macro-for-default-ioprio-settings.patch
      + linux-5.15-compat-ioprio/ioprio-normalize-io-priority-values-in-configuration.patch
      + linux-5.15-compat-ioprio/core-normalize-ioprio-values-we-acquire-from-kernel.patch
      + test-also-show-the-memory-pressure-of-testchill.service.patch
      + test-make-test-55-oomd-less-flaky.patch
      + lp1964494-network-do-not-enable-IPv4-ACD-for-IPv4-link-local-a.patch
      + lp1966381-oomd-calculate-used-memory-with-MemAvailable-instead-of-M.patch
      + lp1926860-hwdb-remove-the-tablet-pad-entry-for-the-UC-Logic-1060N.patch
      + oomd-move-oomctl-to-bindir.patch
      + test-enable-debug-logging-of-systemd-oomd.patch
      + lp1943561/Add-additional-Dell-models-that-require-ACCEL_LOCATION-ba.patch
      + lp1943561/Use-SKU-to-identify-Dell-clamshell-models-for-acceleromet.patch
      + lp1929345/hwdb-Force-release-calculator-key-on-all-HP-OMEN-laptops.patch
      + lp1929345/hwdb-Add-force-release-for-HP-Omen-15-calculator-key.-205.patch
      + sysusers-split-up-systemd.conf.patch
      + hwdb-Add-mic-mute-key-mapping-for-HP-Elite-x360.patch
      + test-check-memory-pressure-more-frequently.patch
      + meson-minor-cleanup.patch
      + units-don-t-install-dbus-org.freedesktop.oom1.service-ali.patch
      + lp1950508-cgroup-check-if-any-controller-is-in-use-as-v1.patch
      + lp1952735-keymap-Add-microphone-mute-keymap-for-Dell-Machine.patch
      + test-tweak-parameters-for-TEST-55-OOMD.patch
      + deny-list-TEST-29-PORTABLE-and-TEST-50-DISSECT.patch
      + lp1955997-unmask-intel-hid-for-HP-machines.patch
      + lp1952733-hwdb-60-keyboard-Update-Dell-Privacy-Micmute-Hotkey-Map.patch
      + Merge-pull-request-20705-from-yuwata-test-oomd-util.patch
    - Refreshed patches
      + debian/Ubuntu-UseDomains-by-default.patch
      + debian/UBUNTU-Support-system-image-read-only-etc.patch
      + Revert-network-if-sys-is-rw-then-udev-should-be-around.patch
      + debian/UBUNTU-src-test-testmount-util.c-Skip-parts-of-test-mount-util-in-LXC.patch
      + lp1950794-Revert-sd-dhcp-do-not-use-detect_container-to-guess-.patch
      + 0001-Revert-tests-add-test-case-for-UMask-BindPaths-combi.patch
  * Drop debian/Ubuntu-core-in-execute-soft-fail-setting-Nice-priority-when.patch.
    This patch...