systemd/245.4-4ubuntu3.18 ADT test failure with linux/5.4.0-128.144

Bug #1991285 reported by Stefan Bader
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
systemd (Ubuntu)
Fix Released

Bug Description

The armhf autopkgtests currently fail in Focal due to this, and failures are not marked as regressions. This limits our ability to catch other regressions on armhf.

[Test Plan]
The boot-and-services autopkgtest should not fail on armhf. Specifically, the test_no_failed test should ignore systemd-remount-fs.service failures on armhf only.

[Where problems could occur]
The fix is limited to the boot-and-services autopkgtest, so there is limited risk. Any bugs would show up in that test.

[Original Description]

This is a scripted bug report about ADT failures while running systemd tests for linux/5.4.0-128.144 on focal. Whether this is caused by the dep8 tests of the tested source or the kernel has yet to be determined.

Testing failed on:

Related branches

Revision history for this message
Stefan Bader (smb) wrote :

To me it looks like the boot-and-services sub-tests got a lot less reliable on armhf in Focal. The previous systemd version had a higher chance of passing those. There always was root-unittests which required multiple attempts. But now from about 10 attempts there was one which had root-unittests only failing and the rest was roughly split between boot-and-services or both failing.

tags: added: kernel-adt-failure
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1991285

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Juerg Haefliger (juergh) wrote :

Log entries from failure:

FAIL: test_no_failed (__main__.ServicesTest)
No failed units
Traceback (most recent call last):
  File "/tmp/autopkgtest.fs4rtb/build.SVB/src/debian/tests/boot-and-services", line 68, in test_no_failed
    self.assertEqual(failed, [])
AssertionError: Lists differ: ['systemd-remount-fs.service loaded failed[41 chars]ems'] != []

First list contains 1 additional elements.
First extra element 0:
'systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems'

+ []
- ['systemd-remount-fs.service loaded failed failed Remount Root and Kernel File '
- 'Systems']

Ran 23 tests in 5.022s

FAILED (failures=1, skipped=6)
autopkgtest [12:03:21]: test boot-and-services: -----------------------]
autopkgtest [12:03:26]: test boot-and-services: - - - - - - - - - - results - - - - - - - - - -
boot-and-services FAIL non-zero exit status 1

tags: added: foundations-triage-discuss
Revision history for this message
Nick Rosbrook (enr0n) wrote :

Here's a bit more information about the service failure:

-- Logs begin at Sun 2022-09-25 00:05:05 UTC, end at Thu 2022-09-29 12:03:20 UTC. --
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd-remount-fs[74]: mount: /: can't find LABEL=cloudimg-rootfs.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd-remount-fs[59]: /bin/mount for / exited with exit status 1.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Sep 29 12:02:57 autopkgtest-lxd-fexenj systemd[1]: Failed to start Remount Root and Kernel File Systems.

This makes me think it's bug 1576341 again. According to the git history, that patch was not carried in Ubuntu past 2019. I will try to find out if we ever attempted to upstream the change.

tags: removed: foundations-triage-discuss
Changed in systemd (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Revision history for this message
Nick Rosbrook (enr0n) wrote :

It appears the last time this worked was around linux 5.4.0-122-generic, e.g. this test run:

Then we start seeing it around, e.g. this test run:

In other words, it started failing with newer kernels but no changes to systemd. I'm not sure this is having any real impact in the wild, but I don't want actual armhf regressions to be lost due to this (britney does not consider armhf failures a regression currently). I think we should just ignore this failed service on armhf only.

Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Focal):
importance: Undecided → Low
status: New → Triaged
Changed in systemd (Ubuntu):
status: Triaged → Invalid
Nick Rosbrook (enr0n)
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote :

To proceed with this SRU, I would like to have some analysis of what makes this an armhf-only failure. Is it to do with the fact that armhf autopkgtests run in a container and others do not? If so, it seems to me that a more correct fix for the test is to detect that we're running in a container instead of querying the arch.

Changed in systemd (Ubuntu Focal):
status: Triaged → Incomplete
Revision history for this message
Nick Rosbrook (enr0n) wrote :

Thanks for reviewing, Steve.

I took a closer look at this. The failure is indeed caused by the container environment, and not something specific to the armhf arch. In particular, the lxd images ship with a non-empty /etc/fstab which contains an entry for a disk label that is not present:

root@autopkgtest-lxd-iameji:/tmp/autopkgtest.DW7ZuH/build.h0Z/src# blkid -o list
device fs_type label mount point UUID
root@autopkgtest-lxd-iameji:/tmp/autopkgtest.DW7ZuH/build.h0Z/src# cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 1

Hence, systemd-remount-fs.service fails with the errors shown in comment #4. I think it would be better to ignore the failure in containers instead of just armhf.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Stefan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Focal):
status: Incomplete → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (systemd/245.4-4ubuntu3.21)

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.21) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.0.9 (armhf)
bilibop/unknown (s390x)
freeipa/unknown (s390x)
fwupd/1.7.9-1~20.04.1 (armhf)
libsoup2.4/2.70.0-1 (s390x)
libusb-1.0/unknown (s390x)
libvirt/unknown (s390x)
linux-aws-5.11/blacklisted (amd64, arm64)
linux-aws-5.13/blacklisted (amd64, arm64)
linux-aws-5.15/5.15.0-1033.37~20.04.1 (arm64)
linux-aws-5.15/unknown (amd64)
linux-aws-5.8/blacklisted (amd64)
linux-azure-5.11/blacklisted (amd64, arm64)
linux-azure-5.13/blacklisted (arm64)
linux-azure-5.8/blacklisted (amd64)
linux-azure-cvm/5.4.0-1105.111+cvm1 (amd64)
linux-gcp-5.11/blacklisted (amd64)
linux-gcp-5.13/blacklisted (amd64)
linux-gcp-5.15/5.15.0-1030.37~20.04.1 (amd64)
linux-gcp-5.15/5.15.0-1031.38~20.04.1 (arm64)
linux-gcp-5.8/blacklisted (amd64)
linux-gke-5.15/5.15.0-1029.34~20.04.1 (amd64, arm64)
linux-gkeop-5.15/5.15.0-1016.21~20.04.1 (amd64)
linux-hwe-5.11/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-hwe-5.13/blacklisted (amd64, arm64, armhf, ppc64el, s390x)
linux-hwe-5.15/5.15.0-69.76~20.04.1 (amd64, armhf)
linux-hwe-5.8/blacklisted (amd64, arm64, ppc64el, s390x)
linux-ibm/5.4.0-1046.51 (amd64)
linux-intel-5.13/blacklisted (amd64)
linux-intel-iotg-5.15/5.15.0-1027.32~20.04.1 (amd64)
linux-oem-5.10/blacklisted (amd64)
linux-oem-5.13/blacklisted (amd64)
linux-oem-5.14/5.14.0-1059.67 (amd64)
linux-oem-5.6/blacklisted (amd64)
linux-oracle-5.11/blacklisted (amd64)
linux-oracle-5.13/blacklisted (amd64, arm64)
munin/2.0.56-1ubuntu1 (arm64, ppc64el)
pdns-recursor/unknown (s390x)
php7.4/unknown (s390x)
polkit-qt-1/unknown (s390x)
puppet/5.5.10-4ubuntu3 (s390x)
tinyssh/unknown (s390x)
udisks2/2.8.4-1ubuntu2 (amd64)
upower/0.99.11-1build2 (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].


Thank you!

Revision history for this message
Nick Rosbrook (enr0n) wrote :

This autopkgtest run for systemd from 245.4-4ubuntu3.21 focal-proposed shows that armhf is passing again:

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 245.4-4ubuntu3.21

systemd (245.4-4ubuntu3.21) focal; urgency=medium

  * udev: avoid NIC renaming race with kernel (LP: #2002445)
    - debian/patches/lp2002445-netlink-do-not-fail-when-new-interface-name-is-already-us.patch
    - debian/patches/lp2002445-netlink-introduce-rtnl_get-delete_link_alternative_names.patch
    - debian/patches/lp2002445-sd-netlink-restore-altname-on-error-in-rtnl_set_link_name.patch
    - debian/patches/lp2002445-udev-attempt-device-rename-even-if-interface-is-up.patch
    - debian/patches/lp2002445-udev-net-allow-new-link-name-as-an-altname-before-renamin.patch
  * test-seccomp: accept ENOSYS from sysctl(2) too (LP: #1933090)
    Thanks to Roxana Nicolescu
    File: debian/patches/lp1933090-test-seccomp-accept-ENOSYS-from-sysctl-2-too.patch
  * debian/test: ignore systemd-remount-fs.service failure in containers (LP: #1991285)
    File: debian/tests/boot-and-services

 -- Nick Rosbrook <email address hidden> Wed, 15 Mar 2023 11:04:15 -0400

Changed in systemd (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.