Android is not automatically validated

[Zach Pfeffer]

There are a few issues across subsystems that are blocking automatic Android validation. We need to identify them and fix or plan how to fix them.

[Fathi Boudra (fboudra)]

Jeremy has submitted a patch to Paul Larson.
Android support for a basic infrastructure is merged in lp:lava

[Fathi Boudra (fboudra)]

Next steps are for Paul. What's missing to get the code running and boot daily the Android images?

[Jeremy Chang (jeremychang)]

One dependency is this bug for the u-boot, https://bugs.launchpad.net/u-boot-linaro/+bug/776856.

After "mmc init", it should be able to load images from mmc card.

[Zach Pfeffer (pfefferz)]

Is this the only issue blocking the merge?

On 26 May 2011 12:27, Jeremy Chang <email address hidden> wrote:
> One dependency is this bug for the u-boot, https://bugs.launchpad.net/u
> -boot-linaro/+bug/776856.
>
> After "mmc init", it should be able to load images from mmc card.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/784836
>
> Title:
>  Identify what's left to do to get automatic Android validation
>  finished
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/linaro-android/+bug/784836/+subscribe
>

[Paul Larson (pwlars)]

Found and worked around one issue with some funny characters messing up the serial log buffer when logcat is called. I saw one instance where it messed up the terminal so badly that expect couldn't cope with it until the board got rebooted. Scary, but haven't been able to reproduce it so far. Is there some reason why logcat does that? Or is it something that 0xbench is sending to the log that contains non-printable characters?

The latest thing I'm looking at is that 0xbench doesn't seem to save any results. When the results get submitted, it's just empty because nothing was saved to the sdcard partition. Do you know of any reason why it might not save results?

[Jeremy Chang (jeremychang)]

logcat is the android's tool to view system log and filter them.

I used the logcat to view the log of 0xbench and monitor a string to recognize that 0xbench finish the benchmark run.
I don't think 0xbench would print "non-printable characters". I would like to reproduce and see what that looks like.

One thing needed to check is if the sdcard partition is mounted by vold. /system/etc/vold.fstab is the key. In vold.fstab, there is the partition number defined inside. also typing mount to check if the sdcard mounted. If android system does not see sdcard mounted, 0xbench will not write the result file after the benchmark run.

[Zach Pfeffer (pfefferz)]

Jeremy, where does 0xbench write its logs?

[Jeremy Chang (jeremychang)]

0xbench will write to /sdcard/lava/results/0xBenchmark.bundle. 0xbench will write the result depending on the sdcard mounted by android system through vold.

It seems I reproduced once the android shell is messed up by running monkey test. The characters displayed are not right as I typed in. After reboot and rerun the test, it's ok again.

[Zach Pfeffer (pfefferz)]

Okay, so when you run the tests the results get written to the card, correct? In addition, the shell is fine until you run monkey, right?

[Jeremy Chang (jeremychang)]

Yes. to me, it's only the u-boot issue left. I need to do "mmc rescan 0" instead of "mmc init" for loading images from mmc.

Besides, regarding to shell messed up issue, it's not easy to reproduce. Though we need to make sure it still can reboot to master image and send the abnormal error log to dashboard.

[Paul Larson (pwlars)]

I think I found my problem:

I/Benchmark( 1314): Completed in 4024ms.
I/Benchmark( 1314): ============================================================
I/Benchmark( 1314):
I/Benchmark( 1314):
I/Benchmark( 1314): Permission denied, maybe SDCARD mounted to PC?

I'll have to reboot to the android image and see if I can determine what's going on with the sdcard partition. I'm sure the above is not right, but perhaps I still don't have it mounting the correct partition.

[Jeremy Chang (jeremychang)]

When you see "Permission denied, maybe SDCARD mounted to PC?", it's not right.

The sdcard is not mounted by vold and from the perspective of android system, there's no sdcard mounted.

vold.fstab is needed being checked if the partition number and sys device path is fit.

[Zach Pfeffer (pfefferz)]

Shouldn't we just write to /data/lava/results/0xBenchmark.bundle?

[Alexander Sack (asac)]

so can we say what is left to get this working or not? is it just the bundle not getting written and the rest is working?

[Paul Larson (pwlars)]

I have this working now. To get it working in the lab though, we need to do the modification to the master images. I sent an email to Dave Pigott to get this going, so tomorrow whenever he gets in, he should be able to do that. Then I need to update the dispatcher on the server with updates to use the new partition layout, and it should work after that. If I have time tonight, I'll also try to go ahead and get metadata support added to the results in android runs, so that we can do better reporting on them.

[Zach Pfeffer (pfefferz)]

Thanks for the great work Paul! Once everything's moving would you make sure the docs are up-to-date and easy to find?

[Paul Larson (pwlars)]

Zach, I sorted out what's going on I think.
/etc/vold.fstab contains
dev_mount sdcard /mnt/sdcard 6 /devices/platform/omap/omap_hsmmc.0/mmc_host/mmc0

This is arguably not the best way to ensure that /sdcard gets mounted in the proper sport, given that it's very platform specific. So much so, in fact, that it doesn't even work on panda in the -leb image (the non-leb builds do, which is what Jeremy seems to have based this on).

I've tried some other things, but I'm having trouble finding the magic sys path to make vold.fstab happy. Any suggestions for what we could use here?
Typically, the sdcard partition will be on /dev/mmcblk0p6, however the preferred way to mount it would be via the fs label "sdcard".

[Zach Pfeffer (pfefferz)]

We just need to change 0xbench to write to /data or to a configurable root. Jim, will the 0xlab guys do this?

[Jim Huang (jserv)]

To Zach,

Extra patch is required to be applied:
    https://gitorious.org/0xbench

However, such modification might not be merged into 0xbench since 0xlab developers would focus on the detailed test items now.

[Zach Pfeffer (pfefferz)]

Jim, would you list some documentation about the new API change?

[Zach Pfeffer (pfefferz)]

This bug has been completed.

The logs can be retrieved via the serial console.

A bug:

https://bugs.launchpad.net/linaro-android/+bug/803566

...has been filed to address writing the results to media.

A BP:

https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-connect-android-build-to-lava

...has been filed to hook the 2 together.

[Zach Pfeffer (pfefferz)]

Although progress was made, Android is still not automatically validated. Will keep this bug open until it is.

[Paul Larson (pwlars)]

I'm a bit confused by the last two comments, but in general, I think it would be good if we could clarify the intent of this bug as it seems a bit vague at the moment, and seems to have shifted purpose several times since it opened. Originally, the bug was aimed at resolving how to get the initial round of Android support that Jeremy did for LAVA working in the validation lab. This was done, with one caveat that the way results were harvested from the SD card will not work reliably on all platforms, or even images. Since then, we have:
* enabled serial logging on all runs to be captured
* changed the dashboard so that it's easy to get at this information
* enabled daily runs of the panda-leb image with 0xbench
* added metadata to the android run to make it easy to find the precise image it was testing
* added a report to the dashboard server showing android runs

Certainly we can all agree that there is plenty more to be done, but there is *always* more to be done :)
However, for purposes of a bug, it would be useful if the completion criteria were clearly indicated. In it's current state, it appears to me that this bug is more of a long-reaching goal of "do more automated android testing".

[Zach Pfeffer (pfefferz)]

Completion criteria:

Each step automatically flows to the next step in:

A daily build automatically pushes a build to the validation system on successful build completion.
The validation system automatically runs a series of 1 or more tests on the build.
The results are automatically saved and presented in both human readable and machine readable form.
The daily build automatically gets an indication that validation is done and automatically updates a build page with the status of the validation and pointer to the results.

[Frans Gifford (fgiff)]

Adding status of BP <https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-connect-android-build-to-lava>

Completion criteria:

Each step automatically flows to the next step in:

[Done] A daily build automatically pushes a build to the validation system on successful build completion.
[Done] The validation system automatically runs a series of 1 or more tests on the build.
[Done] The results are automatically saved and presented in both human readable and machine readable form.
[Postponed] The daily build automatically gets an indication that validation is done and automatically updates a build page with the status of the validation and pointer to the results.

[Frans Gifford (fgiff)]

If nobody objects, I'd like to close this bug since the headline and description "Android is not automatically validated" "There are a few issues across subsystems that are blocking automatic Android validation. We need to identify them and fix or plan how to fix them." are now fixed.

There are still outstanding issues, such as adding links from lava back to android-build, and improving authentication, but I think we need to raise separate bugs to address these.

[Zach Pfeffer (pfefferz)]

Cool. Thanks for closing it.