Test case core dumps, then lava times out without indication

[Mike Holmes]

https://validation.linaro.org/scheduler/job/98097/log_file#L_21_527

The first test run is the no_hz test, this core dumps an lava times out

<LAVA_TEST_RUNNER>: started
 <LAVA_TEST_RUNNER>: looking for installation work in /lava/lava-test-runner.conf-1390059081
 <LAVA_TEST_RUNNER>: save hardware/software context info...
 <LAVA_TEST_RUNNER>: looking for work in /lava/lava-test-runner.conf-1390059081
 <LAVA_TEST_RUNNER>: running 0_NO_HZ_FULL under lava-test-shell...
 <LAVA_SIGNAL_STARTRUN 0_NO_HZ_FULL f412a45b-3f96-4bc6-8d3e-489aba5847b5>
 echo LAVA_ACK
....
....
....
 [ 32.603691] [<800cd78d>] (vfs_write+0x75/0x114) from [<800cdba7>] (SyS_write+0x2f/0x58)
 [ 32.611549] [<800cdba7>] (SyS_write+0x2f/0x58) from [<800182c1>] (ret_fast_syscall+0x1/0x52)
Section 22
22.1
22.2
22.3
22.4
22.5
 lll00<LAVA_DISPATCHER>2014-01-20 01:26:50 AM WARNING: lava_test_shell has timed out
 <LAVA_DISPATCHER>2014-01-20 01:26:50 AM INFO: attempting to access master filesystem 3:/lava
 <LAVA_DISPATCHER>2014-01-20 01:27:00 AM INFO: Booting the system master image. Attempt: 1
 <LAVA_DISPATCHER>2014-01-20 01:27:00 AM INFO: Perform soft reboot the system
 <LAVA_DISPATCHER>2014-01-20 01:29:00 AM INFO: Soft reboot failed: Soft reboot failed
 <LAVA_DISPATCHER>2014-01-20 01:29:00 AM INFO: Perform hard reset on the system

Even though LAVA knows the test was run, and that it times out, it fails to show an indication in the results bundle which has results for the other tests.

lng-arndale02 lava results lava 7 2014-01-20 04:33:35 2014-01-19 20:13:24
lng-arndale02 kvm results kvm 3 1 2014-01-20 04:33:35 2014-01-20 04:31:26
lng-arndale02 ltp results ltp 1206 20 2014-01-20 04:33:35 2014-01-20 03:27:03
lng-arndale02 ltp-realtime results ltp-realtime 8 2 2014-01-20 04:33:35 2014-01-20 01:57:58

[Tyler Baker (tyler-baker)]

Hi Mike,

Here is what I'm thinking:

* When a test times out, LAVA will add comments to each recorded test result that would read "This test timed out, all recorded results are displayed."

[Mike Holmes (mike-holmes)]

Would it look something like the attached ?
So that you know an expected test result is missing ?

On 31 January 2014 11:47, Tyler Baker <email address hidden> wrote:

> Hi Mike,
>
> Here is what I'm thinking:
>
> * When a test times out, LAVA will add comments to each recorded test
> result that would read "This test timed out, all recorded results are
> displayed."
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Project.
> https://bugs.launchpad.net/bugs/1270874
>
> Title:
> Test case core dumps, then lava times out without indication
>
> Status in LAVA Project:
> Confirmed
>
> Bug description:
> https://validation.linaro.org/scheduler/job/98097/log_file#L_21_527
>
> The first test run is the no_hz test, this core dumps an lava times
> out
>
> <LAVA_TEST_RUNNER>: started
> <LAVA_TEST_RUNNER>: looking for installation work in
> /lava/lava-test-runner.conf-1390059081
> <LAVA_TEST_RUNNER>: save hardware/software context info...
> <LAVA_TEST_RUNNER>: looking for work in
> /lava/lava-test-runner.conf-1390059081
> <LAVA_TEST_RUNNER>: running 0_NO_HZ_FULL under lava-test-shell...
> <LAVA_SIGNAL_STARTRUN 0_NO_HZ_FULL f412a45b-3f96-4bc6-8d3e-489aba5847b5>
> echo LAVA_ACK
> ....
> ....
> ....
> [ 32.603691] [<800cd78d>] (vfs_write+0x75/0x114) from [<800cdba7>]
> (SyS_write+0x2f/0x58)
> [ 32.611549] [<800cdba7>] (SyS_write+0x2f/0x58) from [<800182c1>]
> (ret_fast_syscall+0x1/0x52)
> Section 22
> 22.1
> 22.2
> 22.3
> 22.4
> 22.5
> lll00<LAVA_DISPATCHER>2014-01-20 01:26:50 AM WARNING: lava_test_shell
> has timed out
> <LAVA_DISPATCHER>2014-01-20 01:26:50 AM INFO: attempting to access
> master filesystem 3:/lava
> <LAVA_DISPATCHER>2014-01-20 01:27:00 AM INFO: Booting the system master
> image. Attempt: 1
> <LAVA_DISPATCHER>2014-01-20 01:27:00 AM INFO: Perform soft reboot the
> system
> <LAVA_DISPATCHER>2014-01-20 01:29:00 AM INFO: Soft reboot failed: Soft
> reboot failed
> <LAVA_DISPATCHER>2014-01-20 01:29:00 AM INFO: Perform hard reset on the
> system
>
> Even though LAVA knows the test was run, and that it times out, it
> fails to show an indication in the results bundle which has results
> for the other tests.
>
> lng-arndale02 lava results lava 7 2014-01-20
> 04:33:35 2014-01-19 20:13:24
> lng-arndale02 kvm results kvm 3 1 2014-01-20
> 04:33:35 2014-01-20 04:31:26
> lng-arndale02 ltp results ltp 1206 20 2014-01-20
> 04:33:35 2014-01-20 03:27:03
> lng-arndale02 ltp-realtime results ltp-realtime 8 2
> 2014-01-20 04:33:35 2014-01-20 01:57:58
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-project/+bug/1270874/+subscriptions
>

[Tyler Baker (tyler-baker)]

Ah I see. I think there might be two fixes needed.

Situation One:

* A test starts to run, and some test case results are recorded. For what ever reason the test hangs, and then timeouts.

Situation Two:

* A test starts to run, but no test case results are record, as the test case hung before any were produced.

In the case on #1, I think we should display both error messages. One at the test result level (like your example), and at the test case result level. Obviously in case #2 an error message, an error message at the test result level is all that is needed.