Lava Android tests "cts" and "monkey" fail on snowball android build "snowball-ics-gcc46-igloo-stable-blob/#build=309"

Bug #1006215 reported by Abhishek Paliwal
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
LAVA Validation Lab
Invalid
Medium
Yongqin Liu
Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :
Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :

Test: cts
Failed on build: Tests : glmark2 , skia ,v8 ,mmtest, cts, monkey
Failed on build: https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/#build=316

Logs shows tests are not executed. ( http://validation.linaro.org/lava-server/scheduler/job/21339/log_file)
---

LAVA: (stdout) 06-04 20:08:07 I/192.168.1.126:5555: Saved log device_logcat_1233581651924392947.zip
LAVA: (stdout) 06-04 20:08:07 I/192.168.1.126:5555: Saved log host_log_4629086968001444871.zip
LAVA: (stdout) 06-04 20:08:07 I/192.168.1.126:5555: android.tests.appsecurity package complete: Passed 0, Failed 0, Not Executed 6
LAVA: (stdout) 06-04 20:08:07 I/192.168.1.126:5555: Created xml report file at file:///tmp/lava-android-test/tmp1KRNy2/./android-cts/tools/../../android-WARNING: Could not read cpu information
WARNING: Could not read board information
WARNING: Could not read memory information
ERROR: Test execution error: ValidationError: Object has incorrect type (expected string) object_expr='object.test_runs[0].software_context.image.name', schema_expr='schema.properties.test_runs.items.properties.software_context.properties.image.properties.name.type')
cts/repository/results/2012.06.04_20.04.21/testResult.xml
LAVA: (stdout) 06-04 20:08:07 I/192.168.1.126:5555: XML test result file generated at 2012.06.04_20.04.21. Passed 1, Failed 0, Not Executed 17150
----

Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :
Andy Doan (doanac)
Changed in lava-lab:
milestone: none → 2012.06
Yongqin Liu (liuyq0307)
Changed in lava-lab:
assignee: nobody → Yongqin Liu (liuyq0307)
Yongqin Liu (liuyq0307)
Changed in lava-lab:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Yongqin Liu (liuyq0307) wrote :

About the above failures, the confirmation result as following:

Build ID: 309
LAVA Job ID: 20819
Board Used: Snowball03
Problems:
1. When running the CTS test, the android hanged up.
2. When running monkey test under the hanged up state, the reboot will failed because there is no "Hit any key to stop autoboot" displayed.

Build ID: 310
LAVA Job ID: 20896
Board Used: Snowball03
Problems:
1. When running the glmark2 test, the android hanged up
2. When running next test under the hanged up state, the reboot will failed because there is no "Hit any key to stop autoboot" displayed.

Build ID: 316
LAVA Job ID: 21339
Board Used: Snowball01
Problems:
1. When running the CTS test, the android hanged up

Build ID: 318
LAVA Job ID: 21339
Board Used: Snowball01
Problems:
1. When running the mmtest test, the android crashed
2. When running the CTS test, the android hanged up.
3. When running monkey test under the hanged up state, the reboot will failed because there is no "Hit any key to stop autoboot" displayed.

So the mainly problems should be:
1. The android will hang up when running some test, especially when run CTS.
    But not know if it is the CTS test make it hang up.
2. Then reboot under hang up state will failed in most cases. Because there is no "Hit any key to stop autoboot" displayed when hard reset

Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :
Revision history for this message
Yongqin Liu (liuyq0307) wrote :

BUILD ID: snowball-ics-gcc46-igloo-stable-blob/#build=323
JOB ID: 22105
Board Used: snowball03
Problems:
1. When running the CTS test, the android hanged up.

Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :

cts test also failed on:
https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/#build=327

Logs from: http://validation.linaro.org/lava-server/scheduler/job/22417/log_file
---------
WARNING: Could not read board information
WARNING: Could not read memory information
ERROR: Test execution error: The result (cts1339767407.0) is not existed.
9826407434642834.zip
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: android.tests.appsecurity package complete: Passed 0, Failed 0, Not Executed 6
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: Created xml report file at file:///tmp/lava-android-test/tmpk8nMgM/./android-cts/tools/../../android-cts/repository/results/2012.06.15_13.39.55/testResult.xml
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: XML test result file generated at 2012.06.15_13.39.55. Passed 0, Failed 0, Not Executed 17151
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: Time: 1m 15s
LAVA: (stdout) 06-15 13:41:13 W/DeviceManager: Freed device 192.168.1.88:5555 is unavailable. Removing from use.
LAVA: (stdout) 06-15 13:41:13 I/CommandScheduler: All done
LAVA: (stdout) Saved log to /tmp/tradefed_global_log_2564213984193069375.txt
ANDROID TEST RUN COMPLETE: Result id is 'cts1339767407.0'

<LAVA_DISPATCHER>2012-06-15 02:02:32 PM INFO: Execute adb command on host: adb disconnect 192.168.1.88

No such device 192.168.1.88:5555

<LAVA_DISPATCHER>2012-06-15 02:02:33 PM WARNING: [ACTION-E] lava_android_test_run is finished with error (Failed to run test case(cts) on device(192.168.1.88:5555) with return value: 1).

---------

Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :
Revision history for this message
Andy Doan (doanac) wrote :

For comment 7 about the igloo failure. I think a couple of lines should be include just before the log included in that comment:

LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: -----------------------------------------
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: Test package android.tests.appsecurity started
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: -----------------------------------------
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: Saved log device_logcat_1459837433632431998.zip
LAVA: (stdout) 06-15 13:41:11 I/192.168.1.88:5555: Saved log host_log_505WARNING: Could not read cpu information
WARNING: Could not read board information
WARNING: Could not read memory information
ERROR: Test execution error: The result (cts1339767407.0) is not existed.
9826407434642834.zip

I think the CTS may have crashed or something while executing android.tests.appsecurity started. I also noticed that in all the other CTS tests there were lines like:

  android.core.vm-tests-tf package complete: Passed 0, Failed 0, Not Executed 3098

So it looks like CTS wasn't executing anything?

Revision history for this message
Yongqin Liu (liuyq0307) wrote :

The "Not Executed" mainly because the adb connection lost during the CTS.
but why the adb connection is lost should be investigated in android team.
or in the CTS test program.

And in the CTS update BP I am doing now, I add some source for trying to reconnect the adb connection and continue the CTS test.
and print the log of /proc/kmsg and logcat for the investigation about the error.

Actually, because the CTS will run for several hours, there are many cases will cause the adb connection disconnected.
like the memory leak in the program or test program, or network stability, or android crash.
But we need to do more investigation on the CTS test from android side, not from lava side I think.

Revision history for this message
Yongqin Liu (liuyq0307) wrote :

For comment #8, the reason is the same as before:
2. When running monkey test under the hanged up state, the reboot failed because there is no "Hit any key to stop autoboot" displayed.

Fathi Boudra (fboudra)
Changed in lava-lab:
milestone: 2012.06 → 2012.07
Revision history for this message
Abhishek Paliwal (abhishek-paliwal) wrote :
Revision history for this message
Yongqin Liu (liuyq0307) wrote :

The same as Job #21339

Board Used: Snowball01
Problems:
1. When running the mmtest test, the android crashed
2. When running the CTS test, the android hanged up.
3. When running monkey test under the hanged up state, the reboot will failed because there is no "Hit any key to stop autoboot" displayed.

Revision history for this message
Dave Pigott (dpigott) wrote : Re: [Bug 1006215] Lava Android tests "cts" and "monkey" fail on snowball android build "snowball-ics-gcc46-igloo-stable-blob/#build=309"

On 3 Jul 2012, at 08:28, Yongqin Liu wrote:

> The same as Job #21339
>
> Board Used: Snowball01
> Problems:
> 1. When running the mmtest test, the android crashed
> 2. When running the CTS test, the android hanged up.
> 3. When running monkey test under the hanged up state, the reboot will failed because there is no "Hit any key to stop autoboot" displayed.

This is (mostly) because of the sleep 3 in the conmux config. I've changed it to sleep 1 and restarted the daemons in the hope that we catch it more often. The real solution is to stop using conmux for snowball, which I'm working on as a background task.

Dave

>
> --
> You received this bug notification because you are a member of Linaro
> Validation Team, which is subscribed to LAVA Validation Lab.
> https://bugs.launchpad.net/bugs/1006215
>
> Title:
> Lava Android tests "cts" and "monkey" fail on snowball android build
> "snowball-ics-gcc46-igloo-stable-blob/#build=309"
>
> Status in LAVA Validation Lab:
> Confirmed
>
> Bug description:
> Description:
>
> On Android build:
> https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/#build=309
>
> Following LAVA tests failed :
>
> lava_android_test_run (monkey)
> Logs:
> http://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/android-daily/bundles/8001c12e9609f08bca6ef6141378bc46fe537577/6330f88a-a970-11e1-9729-68b599be548c/result/12/
>
> lava_android_test_run (cts)
> Logs:
> http://validation.linaro.org/lava-server/dashboard/streams/private/team/linaro/android-daily/bundles/8001c12e9609f08bca6ef6141378bc46fe537577/6330f88a-a970-11e1-9729-68b599be548c/result/11/
>
> jobid:
> http://validation.linaro.org/lava-server/scheduler/job/20819
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/lava-lab/+bug/1006215/+subscriptions

Revision history for this message
Amit Khare (amit-khare) wrote :

Monkey test failed on build: #351
android-build - https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/#build=351
Job-id : http://validation.linaro.org/lava-server/scheduler/job/24266
Log-file: http://validation.linaro.org/lava-server/scheduler/job/24266/log_file

From Logs:

[ 3.632781] init: cannot find '/system/bin/sh', disabling 'console'
[ 3.639129] init: cannot find '/system/bin/servicemanager', disabling 'servicemanager'
[ 3.647155] init: cannot find '/system/bin/vold', disabling 'vold'
[ 3.653381] init: cannot find '/system/bin/netd', disabling 'netd'
[ 3.659606] init: cannot find '/system/bin/debuggerd', disabling 'debuggerd'
[ 3.666687] init: cannot find '/system/bin/rild', disabling 'ril-daemon'
[ 3.673522] init: cannot find '/system/bin/surfaceflinger', disabling 'surfaceflinger'
[ 3.681488] init: cannot find '/system/bin/app_process', disabling 'zygote'
[ 3.688476] mmc1: host does not support reading read-only switch. assuming write-enable.
[ 3.688537] init: cannot find '/system/bin/drmserver', disabling 'drm'
[ 3.688537] init: cannot find '/system/bin/mediaserver', disabling 'media'
[ 3.688568] init: cannot find '/system/bin/dbus-daemon', disabling 'dbus'
[ 3.688598] init: cannot find '/system/bin/installd', disabling 'installd'
[ 3.688598] init: cannot find '/system/etc/install-recovery.sh', disabling 'flash_recovery'
[ 3.688629] init: cannot find '/system/bin/keystore', disabling 'keystore'
[ 3.688659] init: cannot find '/system/bin/gatord', disabling 'gatord'
[ 3.688690] init: cannot find '/system/bin/hciattach', disabling 'hciattach'
[ 3.688690] init: cannot find '/system/bin/logwrapper', disabling 'admsrv'
[ 3.688720] init: cannot find '/system/bin/lbsd', disabling 'lbsd'
[ 3.688812] init: property (null) has no value for writing to /sys/class/android_usb/android0/iProduct
[ 3.690155] init: cannot find '/system/bin/sh', disabling 'console'
[ 3.692810] adb_open
[ 3.692871] rndis_function_bind_config MAC: 00:00:00:00:00:00
[ 3.692932] gadget: using random self ethernet address
[ 3.692932] gadget: using random host ethernet address
[ 3.694152] rndis0: MAC 82:2d:68:76:9f:e3
[ 3.694152] rndis0: HOST MAC 92:7c:5c:12:91:52
[ 3.694183] adb_bind_config
[ 3.810943] mmc1: new SDHC card at address b368
[ 3.816131] mmcblk1: mmc1:b368 LEXAR 14.9 GiB
[ 3.824066] mmcblk1: p1 p2 p3 p4 < p5 p6 >
[ 3.987548] ab8500-charger ab8500-charger.0: USB Type - Charging not allowed
[ 8.519805] av8100_hdmi av8100_hdmi.3: HDMI display probed
[ 10.360748] av8100_hdmi av8100_hdmi.3: Framebuffer created (av8100_hdmi)
[ 10.367645] init_dispdev: name=disp0 w=1280, h=720, fmt=0, stride=2560
[ 10.374572] av8100_hdmi av8100_hdmi.3: Disp dev created for (av8100_hdmi)

Revision history for this message
Andy Doan (doanac) wrote : Re: [Bug 1006215] Re: Lava Android tests "cts" and "monkey" fail on snowball android build "snowball-ics-gcc46-igloo-stable-blob/#build=309"

On 07/05/2012 01:44 AM, Amit Khare wrote:
> Monkey test failed on build: #351
> android-build -https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/#build=351
> Job-id :http://validation.linaro.org/lava-server/scheduler/job/24266
> Log-file:http://validation.linaro.org/lava-server/scheduler/job/24266/log_file
>
>>From Logs:
>
> [ 3.632781] init: cannot find '/system/bin/sh', disabling 'console'

This is actually a different type of bug we see in the lab periodically.
We don't yet know what the cause of it is.

Andy Doan (doanac)
Changed in lava-lab:
milestone: 2012.07 → 2012.08
Fathi Boudra (fboudra)
Changed in lava-lab:
status: Confirmed → Invalid
milestone: 2012.08 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.