autopkgtests failing in groovy

[Christian Ehrhardt ]

The three most recent tests all failed, but without a clear reason to do so by package changes.

Two of the cases are actually guest kernels getting stuck on boot:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200911_002132_17691@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200910_111212_d54fc@/log.gz

[ 3461.006125] watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:0:7]
[ 3489.006134] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3517.006214] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3545.006160] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
[ 3573.006151] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u2:0:7]
qemu-system-x86_64: terminating on signal 15 from pid 10120 (timeout)
xkvm cleanup killing kvm pid '10146'
xkvm returned 124 in 3600s

One other case doesn't even get that far, it fails on modifying the qemu image:
CalledProcessError: Command '['patch-image', '/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/disk.img', '--kernel=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/kernel', '--initrd=/tmp/autopkgtest.z8gwmE/build.K76/src/debian/tests/groovy.d/initrd']' returned non-zero exit status 1

We might consider the upper two flaky until we know more, but the latter just happened with the new qemu which is suspicious. Yet that new qemu had no changes to qemu-img (except being a new build with a potentially new toolchain).

Yet OTOH the test runs on all architectures and they all worked:
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/s390x
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/arm64
https://autopkgtest.ubuntu.com/packages/o/open-iscsi/groovy/ppc64el

Running the same repro locally in autopkgtest with & without the new qemu works just fine even through multiple re-runs. So it might (tm) resolve on autopkgtest infra as well, not sure.
I'm triggering retries now and we will see later on.

[Christian Ehrhardt  (paelzer)]

So yeah, it resolved by itself in the retry
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/amd64/o/open-iscsi/20200914_062959_d9dcc@/log.gz

Still unsure what that was, the bug might serve as documentation if it appears again - but otherwise has no immediate action to work on left.