Kernel panics / testbed failures in scalingstack

[Dimitri John Ledkov]

For example:

http://autopkgtest.ubuntu.com/packages/s/systemd/cosmic/i386

239-7ubuntu7 dbus/1.12.10-1ubuntu2 2018-09-07 01:51:58 UTC 3h 48m 57s jbicha fail log   artifacts   ♻

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/i386/s/systemd/20180906_071512_4d358@/log.gz

Has a kernel panic, and it ends with a testbed failure. Reruns help.

The tell tell sign is 3+/4+ plus runs, failure, test timeout, and kernel panic in the log (ie. grep for "cut here" and "end trace")

other examples

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/i386/s/systemd/20180907_015158_a1636@/log.gz

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/i386/s/systemd/20180904_182744_cf453@/log.gz

I've seen this for small packages too, where a "retry" fixes everything.

[Iain Lane (laney)]

this sounds like a kernel bug - or do you think there is something for autopkgtest / autopkgtest-cloud to do?

[Dimitri John Ledkov (xnox)]

The kernel itself is fine on retry or when the cloud is quiet. Thus it requires logs from the nova to check that the host/vm was stable at the time. It feels like an over-commit issue with cloud itself faulting the kernel in the VM.