systemd/245.4-4ubuntu3.18 ADT test failure with linux/5.4.0-128.144

[Stefan Bader]

[Impact]
The armhf autopkgtests currently fail in Focal due to this, and failures are not marked as regressions. This limits our ability to catch other regressions on armhf.

[Test Plan]
The boot-and-services autopkgtest should not fail on armhf. Specifically, the test_no_failed test should ignore systemd-remount-fs.service failures on armhf only.

[Where problems could occur]
The fix is limited to the boot-and-services autopkgtest, so there is limited risk. Any bugs would show up in that test.

[Original Description]

This is a scripted bug report about ADT failures while running systemd tests for linux/5.4.0-128.144 on focal. Whether this is caused by the dep8 tests of the tested source or the kernel has yet to be determined.

Testing failed on:
    armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/armhf/s/systemd/20220929_122501_863cf@/log.gz

[Stefan Bader (smb)]

To me it looks like the boot-and-services sub-tests got a lot less reliable on armhf in Focal. The previous systemd version had a higher chance of passing those. There always was root-unittests which required multiple attempts. But now from about 10 attempts there was one which had root-unittests only failing and the rest was roughly split between boot-and-services or both failing.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1991285

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Juerg Haefliger (juergh)]

Log entries from failure:

======================================================================
FAIL: test_no_failed (__main__.ServicesTest)
No failed units
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.fs4rtb/build.SVB/src/debian/tests/boot-and-services", line 68, in test_no_failed
    self.assertEqual(failed, [])
AssertionError: Lists differ: ['systemd-remount-fs.service loaded failed[41 chars]ems'] != []

First list contains 1 additional elements.
First extra element 0:
'systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems'

+ []
- ['systemd-remount-fs.service loaded failed failed Remount Root and Kernel File '
- 'Systems']

----------------------------------------------------------------------
Ran 23 tests in 5.022s

FAILED (failures=1, skipped=6)
autopkgtest [12:03:21]: test boot-and-services: -----------------------]
autopkgtest [12:03:26]: test boot-and-services: - - - - - - - - - - results - - - - - - - - - -
boot-and-services FAIL non-zero exit status 1

[Nick Rosbrook (enr0n)]

Here's a bit more information about the service failure:

-- Logs begin at Sun 2022-09-25 00:05:05 UTC, end at Thu 2022-09-29 12:03:20 UTC. --
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd-remount-fs[74]: mount: /: can't find LABEL=cloudimg-rootfs.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd-remount-fs[59]: /bin/mount for / exited with exit status 1.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: Failed to start Remount Root and Kernel File Systems.
FAIL

This makes me think it's bug 1576341 again. According to the git history, that patch was not carried in Ubuntu past 2019. I will try to find out if we ever attempted to upstream the change.

[Nick Rosbrook (enr0n)]

It appears the last time this worked was around linux 5.4.0-122-generic, e.g. this test run: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/armhf/s/systemd/20220907_155827_ae562@/log.gz.

Then we start seeing it around 5.4.0.128.129, e.g. this test run: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/armhf/s/systemd/20220922_145908_d6960@/log.gz.

In other words, it started failing with newer kernels but no changes to systemd. I'm not sure this is having any real impact in the wild, but I don't want actual armhf regressions to be lost due to this (britney does not consider armhf failures a regression currently). I think we should just ignore this failed service on armhf only.

[Steve Langasek (vorlon)]

To proceed with this SRU, I would like to have some analysis of what makes this an armhf-only failure. Is it to do with the fact that armhf autopkgtests run in a container and others do not? If so, it seems to me that a more correct fix for the test is to detect that we're running in a container instead of querying the arch.

[Nick Rosbrook (enr0n)]

Thanks for reviewing, Steve.

I took a closer look at this. The failure is indeed caused by the container environment, and not something specific to the armhf arch. In particular, the lxd images ship with a non-empty /etc/fstab which contains an entry for a disk label that is not present:

root@autopkgtest-lxd-iameji:/tmp/autopkgtest.DW7ZuH/build.h0Z/src# blkid -o list
device fs_type label mount point UUID
-------------------------------------------------------------------------------------------------------
root@autopkgtest-lxd-iameji:/tmp/autopkgtest.DW7ZuH/build.h0Z/src# cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 1

Hence, systemd-remount-fs.service fails with the errors shown in comment #4. I think it would be better to ignore the failure in containers instead of just armhf.

[Łukasz Zemczak (sil2100)]

Hello Stefan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.21) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.0.9 (armhf)
bilibop/unknown (s390x)
freeipa/unknown (s390x)
fwupd/1.7.9-1~20.04.1 (armhf)
libsoup2.4/2.70.0-1 (s390x)
libusb-1.0/unknown (s390x)
libvirt/unknown (s390x)
linux-aws-5.11/blacklisted (amd64, arm64)
linux-aws-5.13/blacklisted (amd64, arm64)
linux-aws-5.15/5.15.0-1033.37~20.04.1 (arm64)
linux-aws-5.15/unknown (amd64)
linux-aws-5.8/blacklisted (amd64)
linux-azure-5.11/blacklisted (amd64, arm64)
linux-azure-5.13/blacklisted (arm64)
linux-azure-5.8/blacklisted (amd64)
linux-azure-cvm/5.4.0-1105.111+cvm1 (amd64)
linux-gcp-5.11/blacklisted (amd64)
linux-gcp-5.13/blacklisted (amd64)
linux-gcp-5.15/5.15.0-1030.37~20.04.1 (amd64)
linux-gcp-5.15/5.15.0-1031.38~20.04.1 (arm64)
linux-gcp-5.8/blacklisted (amd64)
linux-gke-5.15/5.15.0-1029.34~20.04.1 (amd64, arm64)
linux-gkeop-5.15/5.15.0-1016.21~20.04.1 (amd64)
linux-hwe-5.11/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-hwe-5.13/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-hwe-5.15/5.15.0-69.76~20.04.1 (amd64, armhf)
linux-hwe-5.8/blacklisted (amd64, arm64, ppc64el, s390x)
linux-ibm/5.4.0-1046.51 (amd64)
linux-intel-5.13/blacklisted (amd64)
linux-intel-iotg-5.15/5.15.0-1027.32~20.04.1 (amd64)
linux-oem-5.10/blacklisted (amd64)
linux-oem-5.13/blacklisted (amd64)
linux-oem-5.14/5.14.0-1059.67 (amd64)
linux-oem-5.6/blacklisted (amd64)
linux-oracle-5.11/blacklisted (amd64)
linux-oracle-5.13/blacklisted (amd64, arm64)
munin/2.0.56-1ubuntu1 (arm64, ppc64el)
pdns-recursor/unknown (s390x)
php7.4/unknown (s390x)
polkit-qt-1/unknown (s390x)
puppet/5.5.10-4ubuntu3 (s390x)
tinyssh/unknown (s390x)
udisks2/2.8.4-1ubuntu2 (amd64)
upower/0.99.11-1build2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Nick Rosbrook (enr0n)]

This autopkgtest run for systemd from 245.4-4ubuntu3.21 focal-proposed shows that armhf is passing again: https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/armhf/s/systemd/20230327_104352_bc3ad@/log.gz.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.21

---------------
systemd (245.4-4ubuntu3.21) focal; urgency=medium

  * udev: avoid NIC renaming race with kernel (LP: #2002445)
    Files:
    - debian/patches/lp2002445-netlink-do-not-fail-when-new-interface-name-is-already-us.patch
    - debian/patches/lp2002445-netlink-introduce-rtnl_get-delete_link_alternative_names.patch
    - debian/patches/lp2002445-sd-netlink-restore-altname-on-error-in-rtnl_set_link_name.patch
    - debian/patches/lp2002445-udev-attempt-device-rename-even-if-interface-is-up.patch
    - debian/patches/lp2002445-udev-net-allow-new-link-name-as-an-altname-before-renamin.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=69ab4a02e828e20ea0ddbd75179324df7a8d1175
  * test-seccomp: accept ENOSYS from sysctl(2) too (LP: #1933090)
    Thanks to Roxana Nicolescu
    File: debian/patches/lp1933090-test-seccomp-accept-ENOSYS-from-sysctl-2-too.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=adaddd1441370ebcdb8bc33d7406b95d85b744f9
  * debian/test: ignore systemd-remount-fs.service failure in containers (LP: #1991285)
    File: debian/tests/boot-and-services
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=264bdc86f1e4dcd10e8d914d095581c54c33199a

 -- Nick Rosbrook <email address hidden> Wed, 15 Mar 2023 11:04:15 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.