testbed auxverb failed with exit code 255 with linux on Groovy ADT failure

[Kelsey Steele]

Testing failed on:
    arm64: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-groovy/groovy/arm64/l/linux/20210308_234307_e716b@/log.gz

Looks to be a flaky test with this error occurring frequently. This is not a regression.

Found a previously reported (expired) bug with the same error:
https://launchpad.net/bugs/1549425

[ 6606.751232] audit: backlog limit Creating nova instance adt-groovy-arm64-linux-20210308-143145 from image adt/ubuntu-groovy-arm64-server-20210308.img (UUID 1cfb02a6-bf88-46bb-a8f1-3c6589cd939e)...
Creating nova instance adt-groovy-arm64-linux-20210308-143145 from image adt/ubuntu-groovy-arm64-server-20210308.img (UUID 1cfb02a6-bf88-46bb-a8f1-3c6589cd939e)...
autopkgtest [23:42:55]: ERROR: testbed failure: testbed auxverb failed with exit code 255

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1918226

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Kleber Sacilotto de Souza (kleber-souza)]

I suspect the "ERROR: testbed failure: testbed auxverb failed with exit code 255" messages are only the symptom and not the cause of the failures.

All the arm64 failures are consistent and the last tests executed are always the following:

[...]
19:05:33 INFO | START ubuntu_kernel_selftests.ftrace ubuntu_kernel_selftests.ftrace timestamp=1600715133 timeout=1800 localtime=Sep 21 19:05:33
[...]
19:06:44 DEBUG| [stdout] # [46] Create/delete multiprobe on kprobe event [PASS]
19:06:46 DEBUG| [stdout] # [47] Kprobe event parser error log check [PASS]
19:06:46 DEBUG| [stdout] # [48] Kretprobe dynamic event with arguments [PASS]
19:06:47 DEBUG| [stdout] # [49] Kretprobe dynamic event with maxactive [PASS]
autopkgtest-virt-ssh [19:26:52]: ------- nova console-log 939a7972-4eb7-4814-a489-046ce015ad8e (adt-groovy-arm64-linux-20200921-123119) ------
[...]

From comment #2 on bug 1549425:

"The auxverb is "ssh" in this case, and it exiting with 255 means that the ssh connection was interrupted."

So I suspect some ftrace test is causing a kernel hang, killing the ssh connection. I don't see anything suspicious on the adt console logs though. We might need to try to reproduce this manually to get some more information.

[Francis Ginther (fginther)]

I also looked at this and compared with older kernels. This test has never progressed past the "Kretprobe dynamic event with maxactive" test on 5.8, but it has on 5.4:

```
18:41:31 DEBUG| [stdout] # [43] Kprobe event parser error log check [PASS]
18:41:32 DEBUG| [stdout] # [44] Kretprobe dynamic event with arguments [PASS]
18:41:33 DEBUG| [stdout] # [45] Kretprobe dynamic event with maxactive [PASS]
18:41:49 DEBUG| [stdout] # [46] Register/unregister many kprobe events [PASS]
18:41:49 DEBUG| [stdout] # [47] Kprobe dynamic event - adding and removing [PASS]
18:41:50 DEBUG| [stdout] # [48] Uprobe event parser error log check [PASS]
18:41:50 DEBUG| [stdout] # [49] test for the preemptirqsoff tracer [UNSUPPORTED]
18:42:32 DEBUG| [stdout] # [50] Meta-selftest [PASS]
```

I suspect that next test is causing a hang and ssh dies. The autopkgtest infrastructure sees this and tries to provide some debugging info (the console log and vm details).

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release