lava boot test for panda has failed since build 314

[Patrik Ryd]

The lava boot test has failed since https://android-build.linaro.org/builds/~linaro-android/panda/#build=314.

[Chao Yang (chaoyang)]

the lava dispatch complains:
<LAVA_DISPATCHER>2011-11-19 08:52:30 AM ERROR: boot_linaro_android_image failed: list index out of range
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/actions/boot_control.py", line 36, in run
    client.boot_linaro_android_image()
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/client.py", line 235, in boot_linaro_android_image
    self.proc._boot(string_to_list(self.config.get('boot_cmds_android')))
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/connection.py", line 109, in _boot
    self.proc.sendline(boot_cmds[0])
IndexError: list index out of range

This issue is found on:
https://android-build.linaro.org/builds/~linaro-android/landing-snowball-11.11-release/#build=3
https://android-build.linaro.org/builds/~linaro-android/staging-origen-11.11-release/#build=3
https://android-build.linaro.org/builds/~linaro-android/staging-imx53-11.11-release/#build=2
https://android-build.linaro.org/builds/~linaro-android/panda-11.11-release/#build=5
https://android-build.linaro.org/builds/~linaro-android/beagle-11.11-release/#build=3

[Yongqin Liu (liuyq0307)]

https://android-build.linaro.org/builds/~linaro-android/panda-11.11-release/#build=5 is because of network error:
there is the error information below:

wget -qO- http://192.168.1.10/images/tmp/tmpCLUoCI/system.tar.bz2 |tar --numeric-owner -C /mnt/lava -xjf -
wget -qO- http://192.168.1.10/images/tmp/tmpCLUoCI/syste
m.tar.bz2 |tar --numeric-owner -C /mnt/lava -xjf -

bzip2: Compressed file ends unexpectedly;
 perhaps it is corrupted? *Possible* reason follows.
bzip2: Invalid argument
 Input file = (stdin), output file = (stdout)

[Yongqin Liu (liuyq0307)]

https://android-build.linaro.org/builds/~linaro-android/staging-origen-11.11-release/#build=3 is because we didn't defined boot_cmds_android on origen:

it has error information as below:
<LAVA_DISPATCHER>2011-11-19 08:52:30 AM ERROR: boot_linaro_android_image failed: list index out of range
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/actions/boot_control.py", line 36, in run
    client.boot_linaro_android_image()
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/client.py", line 235, in boot_linaro_android_image
    self.proc._boot(string_to_list(self.config.get('boot_cmds_android')))
  File "/usr/lib/pymodules/python2.7/lava_dispatcher/connection.py", line 109, in _boot
    self.proc.sendline(boot_cmds[0])
IndexError: list index out of range

[Yongqin Liu (liuyq0307)]

https://android-build.linaro.org/builds/~linaro-android/landing-snowball-11.11-release/#build=3 has the same error as bug 893089:
https://bugs.launchpad.net/lava-dispatcher/+bug/893089

it has information as below:

  Press <ENTER> to continue...

-bash: mmc: command not found

root@master:~# cu: Got hangup signal
Connected.
Connected.

[Yongqin Liu (liuyq0307)]

https://android-build.linaro.org/builds/~linaro-android/staging-imx53-11.11-release/#build=2 is because bzip2 is not installed.

there is error information as below:
wget -qO- http://192.168.1.10/images/tmp/tmpxxN_tO/boot.
tar.bz2 |tar --numeric-owner -C /mnt/lava -xjf -
tar (grandchild): bzip2: Cannot exec: No such file or directory
tar (grandchild): Error is not recoverable: exiting now
tar: Child died with signal 13
tar: Error is not recoverable: exiting now

[Yongqin Liu (liuyq0307)]

https://android-build.linaro.org/builds/~linaro-android/beagle-11.11-release/#build=3 is because that the master image does not support ext4 filesystem:
for the error message below:

root@master:~# [rc=0]: mount /dev/disk/by-label/testrootfs /mnt/lava/system
mount /dev/disk/by-label/testrootfs /mnt/lava/system
mount: unknown filesystem type 'ext4'
root@master:~# [rc=32]:

[Yongqin Liu (liuyq0307)]

the bug original "https://android-build.linaro.org/builds/~linaro-android/panda/#build=314." is the same as #5:
for the same error message as below:
root@master:~# [rc=0]: wget -qO- http://192.168.1.10/images/tmp/tmpUdxkqE/system.tar.bz2 |tar --numeric-owner -C /mnt/lava -xjf -
wget -qO- http://192.168.1.10/images/tmp/tmpUdxkqE/syste
m.tar.bz2 |tar --numeric-owner -C /mnt/lava -xjf -
tar (grandchild): bzip2: Cannot exec: No such file or directory
tar (grandchild): Error is not recoverable: exiting now
tar: Child died with signal 13
tar: Error is not recoverable: exiting now

[Michael Hudson-Doyle (mwhudson)]

Looks like we need bzip2 installed on the master images?

[Paul Larson (pwlars)]

wow, there are several things here... I think the original bug has clearly been resolved, since panda test runs are not all failing now. We did discuss today that for some reason, android might take longer than 5 min (??) to get to a shell prompt. I increased the default timeout to 15 min, and ran 5 or 6 jobs through on panda1 and it passed the boot test on all of them, so hopefully it's even more reliable now.

For comment #5, it looks like imx53-01 was missing a package. We *just* added android support fixes for mx53, so we need to try it again and make sure everythings working, but this should be better now and I've fixed the missing package.

For comment #6, beagleXM boards seem to be running a kernel in the master image that does not include support for ext4. I opened https://bugs.launchpad.net/lava-lab/+bug/893368 to track it.

For comment #4, snowball seems to be giving some trouble in general. The STE guy on our team has even had a lot of trouble getting current images working on it. We have bugs filed for the things we know about, landing team is looking into them. If you can track down more information than what you have listed there, please file a bug on it and include it in there.

Comment #3 - Origen: We needed the recently landed changes for supporting android on boards with partition offsets, which we upgraded to over the weekend. I spent some time working on it today, pushed a few more fixes, and got a successful run of android on origen

Did I miss anything here? If so, please either point out how it is the original intent of the bug and you still see it not working -or- consider opening a new bug if it's one of the other many issues tracked here. Also, for the sake of our sanity, please avoid adding unrelated bugs to a bug report. A new bug report is almost always preferred. :)

Thanks!

[Chao Yang (chaoyang)]

LAVA's failure in booting images results from different reasons. It would be better to track each of them separately. All the problem found above can be tracked in the following 4 bugs.

https://bugs.launchpad.net/lava-android-test/+bug/893439
https://bugs.launchpad.net/lava-android-test/+bug/893442
https://bugs.launchpad.net/lava-dispatcher/+bug/893089
https://bugs.launchpad.net/lava-lab/+bug/893368

[Patrik Ryd (patrik-ryd)]

The original problem has been solved and the similar one that popped for the 11.11 RC2 builds were solved by changing a timeout value as mentioned above by Paul. I will set this to fix committed.