lava-test-shell crashes when commands time out and aborts the entire job

[vishal]

Here is the job definition:
https://validation.linaro.org/lava-server/scheduler/job/48387/definition

The lava-test-shell fails in the job with a timeout and the next command "lava-android-test" doesn't get executed. Here are the result bundles and the complete log:

https://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/android-daily/bundles/e14c21059628131a3f3f4be1924e6f76cd3fde87/eaebc2de-7ac9-11e2-a9d7-fa163e618683/

https://validation.linaro.org/lava-server/scheduler/job/48387/log_file

[Antonio Terceiro (terceiro)]

Hi Vishal

lava-test-shell timed out because the boot failed: the shell prompt never appeared. This is much probably a problem with the build.

When the boot fails, lava-test-shell finishes with a critical error, aborting the job. The next action wouldn't be able to run anyway.

[vishal (vishalbhoj)]

Hi Antonio,

The booting of the image does complete successfully. Here is the result bundle:
https://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/android-daily/bundles/e14c21059628131a3f3f4be1924e6f76cd3fde87/eaebc2de-7ac9-11e2-a9d7-fa163e618683/

It does reach to a shell prompt and then starts executing lava-test-runner. Here is the section from the logs:
"root@linaro# /data/lava/bin/lava-test-runner"

The test execution in lava-test-shell may timeout due to buggy test itself and marking this as a critical error and aborting the job doesn't seem to be the right thing to do.

[vishal (vishalbhoj)]

Here is some relevant discussion from IRC. There are few other cases that needs to be handled as part of cleanup:

<bhoj> terceiro, I wanted to understand how the execution is handled. I am aware that the device is booted between every lava-test-shell call.
<bhoj> How are things structured when we have a mix of lava-test-shell and lava-android-test ?
<terceiro> bhoj: re your last comment - you are right - I misread the log
<terceiro> bhoj: bhoj after lava-test-shell finishes, it leaves the board alone, so the lava-android-test action should work fine
<terceiro> that is indeed some bug that we have to investigate
<bhoj> terceiro, okay.
<terceiro> I think because of the timeout, some result file that was assumed to exist ended up not being created
<bhoj> terceiro, okay. May be as part of cleanup at timeout the running process needs to be killed so that it doesn't affect the steps which follow.
<terceiro> bhoj: yeah

[Nicholas Schutt (nick-schutt)]

Added try/except signaling to lava_test_shell.py and job.py to catch a LavaTestShellTimeout, report an error for the lava_test_shell step, and continue.

Proposed for merge:

  lp:~nick-schutt/lava-dispatcher/nicks-1130814