Lava : vexpress-tc2-01 and vexpress-tc2-benchmark jobs are running incomplete due to various reasons

[Naresh Kamboju]

Daily big.LITTLE Android builds are running on LAVA TC2.
The amount of job complete ration is 20 percentage.
most of the jobs exit in the middle of the execution.

At this point of time I do not have a specific reason for incomplete executions. This issue needs to be investigated.

Following jobs are incomplete

vexpress-tc2-benchmark:
http://validation.linaro.org/lava-server/scheduler/job/58908

vexpress-tc2-01:
http://validation.linaro.org/lava-server/scheduler/job/58573
http://validation.linaro.org/lava-server/scheduler/job/58854
http://validation.linaro.org/lava-server/scheduler/job/58721
http://validation.linaro.org/lava-server/scheduler/job/58081
http://validation.linaro.org/lava-server/scheduler/job/58037
http://validation.linaro.org/lava-server/scheduler/job/57887
http://validation.linaro.org/lava-server/scheduler/job/57630
http://validation.linaro.org/lava-server/scheduler/job/57500

[vishal (vishalbhoj)]

vexpress-tc2-benchmark:
http://validation.linaro.org/lava-server/scheduler/job/58908 was cancelled by me.

vexpress-tc2-01:
http://validation.linaro.org/lava-server/scheduler/job/58573/log_file failed in UEFI
http://validation.linaro.org/lava-server/scheduler/job/58854/log_file failed in UEFI
http://validation.linaro.org/lava-server/scheduler/job/58721/log_file trying to boot into master image
http://validation.linaro.org/lava-server/scheduler/job/58081/log_file trying to boot into master image
http://validation.linaro.org/lava-server/scheduler/job/58037/log_file failed to get bootoader prompt:
spawn telnet pdu09
telnet: could not resolve pdu09/telnet:

^^^ seems to be LAVA failure
http://validation.linaro.org/lava-server/scheduler/job/57887/log_file final result submission failed.
http://validation.linaro.org/lava-server/scheduler/job/57630 failed in workbench
http://validation.linaro.org/lava-server/scheduler/job/57500/log_file failed in workbench.

Currently the critical issue seems to be the failure to boot to android from UEFI and getting stuck in the master image. These need to be resolved. Adding the ARM LT to comment on the UEFI errors we see from UEFI.

[Ryan Harkin (ryanharkin)]

The UEFI failures are down to this bug:

UEFI: TC2: booting from A7 core causes boot failures due to MMC/SD timeouts
https://bugs.launchpad.net/linaro-uefi/+bug/1179461

[Amit Kucheria (amitk)]

Adding a bug to linaro-uefi and assigning to Ryan since I can't find a way to just make this bug dependent on bug 1179461 he pointed to above.

[Matthew Hart (matthew-hart)]

I've raised a bug in Lava Dispatcher on the jobs above that failed to intercept the bootloader prompt

TC2 boards sometimes fail to intercept bootloader
https://bugs.launchpad.net/lava-dispatcher/+bug/1194088

[Naresh Kamboju (naresh-kamboju)]

Following job failed to clone workload-automation.git

http://validation.linaro.org/lava-server/scheduler/job/59037/log_file

<LAVA_DISPATCHER>2013-06-22 08:22:43 AM INFO: Execute command on host: timeout 1800s lava-android-test run hostshell-workload -s 192.168.1.38:5555 -o /tmp/tmp2xcwqW/hostshell-workload.2013-06-22T08:22:43Z.bundle -O -c config_mp4.csv -g ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git
ERROR: Test execution error: Run step 'bash /srv/lava<email address hidden>/lava_android_test/test_definitions/hostshells/workload.sh -s 192.168.1.38:5555 -c config_mp4.csv -g ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git' failed. 1 : ["Cloning into 'workload-automation'...\n", 'fatal: protocol error: bad line length character: Welc\n', 'Failed to clone git repository: ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git\n']
LAVA: --Start Operation: Run test (hostshell-workload) on device(192.168.1.38:5555)
LAVA: Begin to execute command: bash /srv/lava<email address hidden>/lava_android_test/test_definitions/hostshells/workload.sh -s 192.168.1.38:5555 -c config_mp4.csv -g ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git
LAVA: (stdout) Cloning into 'workload-automation'...
LAVA: (stdout) fatal: protocol error: bad line length character: Welc
LAVA: (stdout) Failed to clone git repository: ssh://<email address hidden>/srv/linaro-private.git.linaro.org/workload-automation.git
<LAVA_DISPATCHER>2013-06-22 08:22:55 AM INFO: Execute adb command on host: adb disconnect 192.168.1.38:5555
<LAVA_DISPATCHER>2013-06-22 08:22:55 AM WARNING: [ACTION-E] lava_android_test_run is finished with error (Failed to run test case(hostshell-workload) on device(192.168.1.38:5555) with return value: 1).

[Naresh Kamboju (naresh-kamboju)]

Following job failed to clone workload-automation.git
http://validation.linaro.org/lava-server/scheduler/job/59037/log_file

Bug reported on above issue.
https://bugs.launchpad.net/lava-dispatcher/+bug/1194102

[Naresh Kamboju (naresh-kamboju)]

Following job reported "permission denied to clone workload-automation.git"
http://validation.linaro.org/lava-server/scheduler/job/59037/log_file

https://bugs.launchpad.net/lava-dispatcher/+bug/1194104

[vishal (vishalbhoj)]

We don't hit the UEFI bug very often with latest UEFI release. The vexpress-tc2-01 board in the lab needs to be updated to either use latest UEFI binary or update the flashing procedure to flash test UEFI binary from boot tarball similar to vexpress-tc2-benchmark-01 setup( I guess the setup for bootloader testing).

[Naresh Kamboju (naresh-kamboju)]

Following bug also in-complete
This lava job executed 24 iterations of workload automation out of 40 iterations.
http://validation.linaro.org/lava-server/scheduler/job/59672/log_file#entry1314 : ERROR: boot_linaro_android_image failed: The home screen has not displayed

[Naresh Kamboju (naresh-kamboju)]

linking one more bug:
https://bugs.launchpad.net/lava-dispatcher/+bug/1194633

[Naresh Kamboju (naresh-kamboju)]

linking one more bug
UEFI: VE+TC2 on LAVA Timeout waiting for boot prompt
https://bugs.launchpad.net/lava-dispatcher/+bug/1194633

[Alan Bennett (akbennett)]

Closing this issue as there isn't enough specific detail to be effectively triaged. If there is a specific issue, please create a single issue, add the details. (there are already some duplicate issues that we can see).