Comment 14 for bug 1915126

Revision history for this message
Balint Reczey (rbalint) wrote :

@laney Thank you for looking into the problem.

I will mark systemd-fsckd as flaky and look into the root cause later. It used to be flaky before as well.

I have checked the last 10 failed runs from http://autopkgtest.ubuntu.com/packages/systemd/hirsute/amd64 , with the last one being from 2021-02-24 09:15:57 UTC:

$ cat logs
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_091557_d5731@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_044536_8d27f@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_021229_4c845@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210224_013702_2056d@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_232058_6bbff@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_212809_c332e@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_145406_d6565@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_140557_825b0@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_130532_340ec@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-hirsute/hirsute/amd64/s/systemd/20210223_115952_4304b@/log.gz

$ wget -q -i logs
$ for i in log.gz*; do zcat $i | tail -n 40 | grep -A 16 "@@@ summary" | grep -v PASS ; done
autopkgtest [09:15:42]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [04:45:21]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [02:00:12]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [01:23:55]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [23:20:34]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [21:19:41]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [14:53:35]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:53:19]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:03:15]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [11:59:24]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable

As you can see systemd-fsckd has been skipped in 10 out of 10 runs, but "upstream" test timed out in 7 out of the 10 runs.
If you would like to help improving the failure rate caused by the infrastructure please run the systemd autopkgtests on bigger VMs.