missing userspace binder patches for armv8 Android lsk build

[Naresh Kamboju]

I have downloaded the latest build and boot tested with FVP_Base_Cortex-A57x4-A53x4 Model.

Linux kernel version:
Linux version 3.10.24-00485-g425f051 (jenkins-build@ip-10-60-77-38) (gcc version 4.8.3 20131111 (prerelease) (Linaro GCC 4.8-2013.11) ) #1 SMP Tue Dec 17 05:42:21 UTC 2013

ARMv8 LSK Android build:
https://android-build.linaro.org/builds/~linaro-android/armv8-lsk/#build=9

1. UEFI prompt was waiting for fdt.dtb. I have crated soft link fdt.dtb from fvp-base-gicv2-psci.dtb and booted again which is successful
   fdt.dtb -> fvp-base-gicv2-psci.dtb

2. Rootfs is not stable

After couple of minutes below message popped up and the rootfs stopped responding to any keys.

root@vexpress:/ # [ 45.588449] init: critical process 'healthd' exited 4 times in 4 minutes; rebooting into recovery mode
[ 45.747701] SysRq : Emergency Remount R/O
[ 45.752099] EXT4-fs (vda3): re-mounted. Opts: (null)
[ 45.763586] EXT4-fs (vda5): re-mounted. Opts: (null)
[ 45.764521] Emergency Remount complete
[ 45.851134] Restarting system with command 'recovery'.
[ 45.852037] vexpress-sysreg 1c010000.sysreg: error -22
[ 46.852037] vexpress-reset reboot.15: Unable to restart (-22)
[ 46.852081] Reboot failed -- System halted
[ 66.861996] INFO: rcu_sched detected stalls on CPUs/tasks: { 1 2 3} (detected by 0, t=2102 jiffies, g=18446744073709551543, c=18446744073709551542, q=13)
[ 66.862146] Task dump for CPU 1:
[ 66.862187] swapper/1 R running task 0 0 1 0x00000000
[ 66.862260] Call trace:
[ 66.862324] [<ffffffc000084780>] __switch_to+0x74/0x8c
[ 66.862393] [<ffffffc00009c49c>] irq_enter+0x5c/0x7c
[ 66.862473] [<ffffffc00008dcac>] handle_IPI+0xd8/0x134
[ 66.862540] [<ffffffc0000812e4>] gic_handle_irq+0x78/0x7c

Bug updated with dmesg and logcat.
- lsk-armv8-android-bug.log
- logcat-lsk-armv8-android-bug.log

[Mark Brown (broonie)]

Please file a separate bug for the packaging issue, it is much easier to track multiple bugs if they're in separate reports.

We need more diagnostics on what's going on in userspace for this - the issue is that healthd decided it was unhappy with the state of the system and triggered a reboot (which isn't implemented by the emulator). Do we have a logcat or anything showing why healthd was unhappy?

[Naresh Kamboju (naresh-kamboju)]

I have attached logcat output.
logcat-lsk-armv8-android-bug.log

[Mark Brown (broonie)]

According to Vishal this looks like the binder compat patches may be incomplete or missing.

[Naresh Kamboju (naresh-kamboju)]

[Alex Shi wrote] Does this bug fixed in 14.01 lsk?
[Vishal wrote] This is fixed on the test branch. It is still not merged to the stable branch.

[Naresh Kamboju (naresh-kamboju)]

This bug reproduced on ARMv8 LSK build at boot time.

Android Build:
https://android-build.linaro.org/builds/~linaro-android/armv8-lsk-14.03-release/#build=5
http://snapshots.linaro.org/android/~linaro-android/armv8-lsk-14.03-release/5/

[ 46.359880] binder: 1838:1838 ioctl c0186201 fffef8e8 returned -22
[ 46.360982] init: critical process 'healthd' exited 4 times in 4 minutes; rebooting into recovery mode
[ 46.363685] SysRq : Emergency Remount R/O
[ 46.372898] EXT4-fs (vda3): re-mounted. Opts: (null)
[ 46.374646] EXT4-fs (vda5): re-mounted. Opts: (null)
[ 46.375612] Emergency Remount complete
[ 46.487579] Restarting system with command 'recovery'.
[ 46.492777] vexpress-sysreg 1c010000.sysreg: error -22
[ 47.492776] vexpress-reset reboot.15: Unable to restart (-22)
[ 47.492820] Reboot failed -- System halted
[ 67.502739] INFO: rcu_sched detected stalls on CPUs/tasks: { 1 2 3 4 5 6 7} (detected by 0, t=2102 jiffies, g=18446744073709551504, c=18446744073709551503, q=71)
[ 67.502938] Task dump for CPU 1:

[Naresh Kamboju (naresh-kamboju)]

lava_lsk-armv8-14.03_boot_failed log attached.

[Alex Shi (alex-shi)]

Looks like there are 2 issue in this bug.
1, binder ioctl failed,
2, rcu_sched used incorrect and triggered by reboot.

If binder code in kernel used rcu_sched function set, it maybe one problem. But I am idiot of binder. Anyone like to share some info on this?

[Naresh Kamboju (naresh-kamboju)]

Bug reproduced:

LAVA job:
https://validation.linaro.org/scheduler/job/118841/log_file#L_11_26

Android armv8 lsk-pre-merge-ci :
http://snapshots.linaro.org/android//~linaro-android/armv8-lsk-pre-merge-ci/61

Linux kernel version:
Linux version 3.10.33-00012-g1e23152 (jenkins-build@ip-10-152-139-46) (gcc version 4.8.3 20131111 (prerelease) (Linaro GCC 4.8-2013.11) ) #1 SMP Thu Apr 3 06:21:45 UTC 2014

Error log:
--------------
[ 35.028095] binder: 1827:1827 ioctl c0186201 fffef8e8 returned -22
 [ 35.029288] init: critical process 'healthd' exited 4 times in 4 minutes; rebooting into recovery mode
 [ 35.031140] SysRq : Emergency Remount R/O
 [ 35.041103] EXT4-fs (vda3): re-mounted. Opts: (null)
 [ 35.042762] EXT4-fs (vda5): re-mounted. Opts: (null)
 [ 35.043654] Emergency Remount complete
 [ 35.153143] Restarting system with command 'recovery'.
 [ 35.160964] vexpress-sysreg 1c010000.sysreg: error -22
 [ 36.160968] vexpress-reset reboot.15: Unable to restart (-22)
 [ 36.161012] Reboot failed -- System halted
 [ 56.140926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=2102 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
 [ 56.141055] Task dump for CPU 2:
 [ 56.141096] mediaserver R running task 0 1832 1 0x00400000
 [ 56.141170] Call trace:
 [ 56.141234] [<ffffffc00008498c>] __switch_to+0x74/0x8c
 [ 119.190926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=8407 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
 [ 119.191055] Task dump for CPU 2:
 [ 119.191096] mediaserver R running task 0 1832 1 0x00400000
 [ 119.191171] Call trace:
 [ 119.191235] [] __switch_to+0x74/0x8c
[ 182.240926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=14712 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
 [ 182.241055] Task dump for CPU 2:
 [ 182.241096] mediaserver R running task 0 1832 1 0x00400000
 [ 182.241171] Call trace:
 [ 182.241235] [<ffffffc00008498c>] __switch_to+0x74/0x8c

[Botao (botao-sun)]

@Mark @Naresh, a separated bug has just been reported on my side with the same root casue for:

root@vexpress:/ # [ 45.588449] init: critical process 'healthd' exited 4 times in 4

https://bugs.launchpad.net/linaro-android/+bug/1302381

[Alex Shi (alex-shi)]

@Naresh @vishalbhoj

Is this bug fixed?

[vishal (vishalbhoj)]

Yes. This is fixed.