All tests listed in LAVA_TEST_PLAN are not attempted

[Zach Pfeffer]

A typical build will list the following tests:

LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,cts,monkey"

...but if one of the tests crashes, LAVA will not reboot and attempt the other tests. It should do this.

[Yongqin Liu (liuyq0307)]

Could you give a link for us to confirm the problem?

[Michael Hudson-Doyle (mwhudson)]

I too would like to see a job where this happens. By my reading of the code, if we try to execute a lava_android_test_run action when the board is hung we will reboot it.

I can believe that if a lava_android_test_run action hangs the job might exceed the global timeout and be killed without trying any further tests, or that if a test run damages the board in some way such that the test shell is still there but nothing else works. But that seems like a bit of an edge case somehow.

[Michael Hudson-Doyle (mwhudson)]

So an email that got forwarded to me linked to http://validation.linaro.org/lava-server/scheduler/job/20439 -- which is certainly strange. It seems to fall into the "a test run damages the board in some way such that the test shell is still there but nothing else works" category.

[Zach Pfeffer (pfefferz)]

Just to clarify, Andy and I had talked about this issue when I went looking for CTS results. When I looked at a few LAVA runs I saw it execute this test list:

LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,cts,monkey"

...but it would fail on glmark2 and not attempt skia. This bug was meant to track this issue since I think the build should restart the test list on the test after the one that caused it to crash.

[Yongqin Liu (liuyq0307)]

Sorry, still want to know the job url and see the details.

Normally, LAVA will try to run the next test after the previous test failed
even the android image is restarted.
Like the job url mentioned above,
http://validation.linaro.org/lava-server/scheduler/job/20439
Lava had tried to run the test of skia, but it failed because the android
image screen seems not been started correctly.
Like the following log message in the log:

root@linaro: <LAVA_DISPATCHER>2012-05-24 04:28:06 PM WARNING: [ACTION-E]
lava_android_test_run is finished with error (The home screen has not
displayed).
<LAVA_DISPATCHER>2012-05-24 04:28:06 PM INFO: [ACTION-B]
lava_android_test_run is started with {u'test_name': u'skia'}
/srv/lava/instances/production/bin/lava-android-test
Lava failed at action lava_android_test_run with error: The home screen has
not displayed
Traceback (most recent call last):
  File
"/srv/lava/instances/production/local/lib/python2.7/site-packages/lava_dispatcher/job.py",
line 158, in run
    action.run(**params)
  File
"/srv/lava/instances/production/lib/python2.7/site-packages/lava_dispatcher/actions/lava-android-test.py",
line 58, in run
    with self.client.android_tester_session() as session:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File
"/srv/lava/instances/production/local/lib/python2.7/site-packages/lava_dispatcher/client/base.py",
line 353, in android_tester_session
    session.wait_home_screen()
  File
"/srv/lava/instances/production/local/lib/python2.7/site-packages/lava_dispatcher/client/base.py",
line 242, in wait_home_screen
    raise GeneralError('The home screen has not displayed')
GeneralError: The home screen has not displayed

Thanks,
Yongqin Liu

On 5 June 2012 00:20, Zach Pfeffer <email address hidden> wrote:

> Just to clarify, Andy and I had talked about this issue when I went
> looking for CTS results. When I looked at a few LAVA runs I saw it
> execute this test list:
>
> LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,cts,monkey"
>
> ...but it would fail on glmark2 and not attempt skia. This bug was meant
> to track this issue since I think the build should restart the test list
> on the test after the one that caused it to crash.
>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Android Test.
> https://bugs.launchpad.net/bugs/1003726
>
> Title:
> All tests listed in LAVA_TEST_PLAN are not attempted
>
> Status in Lava Android Test Runner:
> New
> Status in Linaro Android:
> New
>
> Bug description:
> A typical build will list the following tests:
>
> LAVA_TEST_PLAN="busybox,0xbench,glmark2,skia,v8,mmtest,cts,monkey"
>
> ...but if one of the tests crashes, LAVA will not reboot and attempt
> the other tests. It should do this.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-android-test/+bug/1003726/+subscriptions
>

[Michael Hudson-Doyle (mwhudson)]

I think we decided via email that we should reboot the board between each lava-android-test invocation, to be on the safe side?

[Abhishek Paliwal (abhishek-paliwal)]

I've observed this issue with Origen build:
https://android-build.linaro.org/builds/~linaro-android/origen-ics-gcc47-samsunglt-stable-blob/#build=45

Job details:
http://validation.linaro.org/lava-server/scheduler/job/21316/log_file

Here mmtest has failed:
After this failure 'cts' and 'monkey' tests are not attempted.

[Milosz Wasilewski (mwasilew)]

LAVA_TEST_PLAN is not used any more. Closing.