$ wget -q -i logs
$ for i in log.gz*; do zcat $i | tail -n 40 | grep -A 16 "@@@ summary" | grep -v PASS ; done
autopkgtest [09:15:42]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [04:45:21]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [02:00:12]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [01:23:55]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [23:20:34]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [21:19:41]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [14:53:35]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:53:19]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:03:15]: @@@@@@@@@@@@@@@@@@@@ summary
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [11:59:24]: @@@@@@@@@@@@@@@@@@@@ summary
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
As you can see systemd-fsckd has been skipped in 10 out of 10 runs, but "upstream" test timed out in 7 out of the 10 runs.
If you would like to help improving the failure rate caused by the infrastructure please run the systemd autopkgtests on bigger VMs.
@laney Thank you for looking into the problem.
I will mark systemd-fsckd as flaky and look into the root cause later. It used to be flaky before as well.
I have checked the last 10 failed runs from http:// autopkgtest. ubuntu. com/packages/ systemd/ hirsute/ amd64 , with the last one being from 2021-02-24 09:15:57 UTC:
$ cat logs /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210224_ 091557_ d5731@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210224_ 044536_ 8d27f@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210224_ 021229_ 4c845@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210224_ 013702_ 2056d@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 232058_ 6bbff@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 212809_ c332e@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 145406_ d6565@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 140557_ 825b0@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 130532_ 340ec@/ log.gz /objectstorage. prodstack4- 5.canonical. com/v1/ AUTH_77e2ada1e7 a84929a74ba3b87 153c0ac/ autopkgtest- hirsute/ hirsute/ amd64/s/ systemd/ 20210223_ 115952_ 4304b@/ log.gz
https:/
https:/
https:/
https:/
https:/
https:/
https:/
https:/
https:/
https:/
$ wget -q -i logs @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary @@@@@@@ @@@@@@ summary
$ for i in log.gz*; do zcat $i | tail -n 40 | grep -A 16 "@@@ summary" | grep -v PASS ; done
autopkgtest [09:15:42]: @@@@@@@
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [04:45:21]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [02:00:12]: @@@@@@@
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [01:23:55]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [23:20:34]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [21:19:41]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [14:53:35]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:53:19]: @@@@@@@
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [13:03:15]: @@@@@@@
upstream FAIL timed out
systemd-fsckd SKIP exit status 77 and marked as skippable
autopkgtest [11:59:24]: @@@@@@@
boot-and-services FAIL non-zero exit status 1
upstream FAIL non-zero exit status 1
systemd-fsckd SKIP exit status 77 and marked as skippable
As you can see systemd-fsckd has been skipped in 10 out of 10 runs, but "upstream" test timed out in 7 out of the 10 runs.
If you would like to help improving the failure rate caused by the infrastructure please run the systemd autopkgtests on bigger VMs.