boot-and-services in tests-in-lxd autopkgtest fails in groovy on s390x

Bug #1895576 reported by Balint Reczey on 2020-09-14
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Undecided
Unassigned

Bug Description

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/s390x/s/systemd/20200903_184028_2bbdf@/log.gz

...
test_rsyslog (__main__.ServicesTest) ... FAIL
test_tmp_cleanup (__main__.ServicesTest) ... ok
test_tmp_mount (__main__.ServicesTest) ... ok
test_udev (__main__.ServicesTest) ... skipped 'udev does not work in containers'

======================================================================
FAIL: test_rsyslog (__main__.ServicesTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.15TVnz/build.ZVh/real-tree/debian/tests/boot-and-services", line 120, in test_rsyslog
    self.assertRegex(log, 'systemd.*Reached target Graphical Interface')
...

Interestingly it does not fail locally on an s390x Focal VM running boot-and-services in LXD.
Kernel:
Linux juju-d7a408-generic-21 5.4.0-14-generic

Also interestingly it is passing with cryptsetup/2:2.3.3-1ubuntu5 glibc/2.32-0ubuntu2.

Thanks for this bug Rbalint.
That the new glibc/cryptsetup worked is odd as nothing on the issue indicates they are related. There was nothing in the queue - for the sake of some test data I have restarted the packages currently blocked on this with all-proposed which will pull in e.g. gblibc/cryptsetup.

Let us check in some hours how that worked out ...

@Rbalint - hi the two most recent tests confirm that it still seems broken.
Maybe the good run was a one-off lucky run ... :-/

I'd ask you to look into that a bit deeper as it will again hold back plenty of things in groovy soon.

Also please do note that while almost all of the recent tests failed - almost all of the packages already migrated. Why is that - there is no test hint in place.
For example gdm3/3.37.90-1ubuntu1 and iptables/1.8.4-3ubuntu3 (but more than just those, policykit-1, dpdk, util-linux, ...) NEVER had a valid test [1], but they migrated to groovy-release.
That seems even more wrong, any idea why/how that happened?

There might be a follow up issue of people using tools that "retry everything that had no good run" but since somehow things magically migrated the recent long retry-lists mostly contain packages that are already migrated.

For the sake of seeing if it might be "blunt retries" and since the s390x test queue is empty I'm retrying it multiple times (hopefully those won't be coalesced).
The last not-yet weird-migrated entry is apparmor.

[1]: https://autopkgtest.ubuntu.com/packages/s/systemd/groovy/s390x

Old hints we only had:

# failing due to multiple reasons LP: #1892358
force-badtest systemd/246-2ubuntu1
# failing due to livecd-rootfs creating broken LXD images LP: #1877078
force-reset-test systemd/246.2-1ubuntu1

But was then removed in
revno: 4988
committer: Steve Langasek <email address hidden>
branch nick: hints-ubuntu
timestamp: Fri 2020-09-11 13:38:38 -0700
message:
  drop unused hints; fix typo in package name

Maybe that rule let the packages pass that I wondered about, but it is unclear what made 246.4-1ubuntu1 itself pass as there was no good run of that either ?!?.

Overall for now it seems to me that we'd have to add:
force-badtest systemd/246.4-1ubuntu1/s390x

The retries were reproducible failing as well.

[09:20] <cpaelzer> rbalint: btw the test on s390x really seems still broken
[09:20] <cpaelzer> we need either a fix or a test hint
[09:21] --> falcojr (~falcojr@184.55.64.152) has joined this channel.
[09:24] <rbalint> cpaelzer, i'm ok with the hint and i've retriggered the tests with cryptsetup and glibc to see if they still help
[09:25] <cpaelzer> ok rbalint, given you are the maintainer and you agree I think I'll prep a hint and once you ack there (for an audit trail) we can push it
[09:27] <rbalint> cpaelzer, thanks, i'm also working on the next systemd point release but it shows regressions so it will take a few days

Here it is:
https://code.launchpad.net/~paelzer/britney/hints-ubuntu-groovy-systemd-s390x-fails/+merge/390809

For now the hint is in groovy, but the next systemd upload needs to resolve it one (fix) or the other (mark this one as flaky) way.

Balint Reczey (rbalint) wrote :

OK, the next upload passes on s390x, so this will likely be resolved.

Balint Reczey (rbalint) wrote :

I can't start s390x VMs to reproduce it for some reason so I'm just disabling this test in the next upload.

Changed in systemd (Ubuntu):
status: New → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers