Boot process can't be completed on "linaro-android-armv8-lsk-pre-merge-ci".

[Botao]

On build "linaro-android-armv8-lsk-pre-merge-ci":

http://snapshots.linaro.org/android/~linaro-android/armv8-lsk-pre-merge-ci/61

Though Bug 1300696 has gone and system now can start to boot, but the boot process can't be completed, and therefore tests are blocked.

The errors are:

https://validation.linaro.org/dashboard/streams/private/team/linaro/android-daily/bundles/1242cfc3868c82e2c563388e1e6963fe143df337/

https://validation.linaro.org/scheduler/job/118841/log_file

https://validation.linaro.org/scheduler/job/118841/log_file#L_11_17

[ 35.029288] init: critical process 'healthd' exited 4 times in 4 minutes; rebooting into recovery mode
[ 35.031140] SysRq : Emergency Remount R/O
[ 35.041103] EXT4-fs (vda3): re-mounted. Opts: (null)
[ 35.042762] EXT4-fs (vda5): re-mounted. Opts: (null)
[ 35.043654] Emergency Remount complete
[ 35.153143] Restarting system with command 'recovery'.
[ 35.160964] vexpress-sysreg 1c010000.sysreg: error -22
[ 36.160968] vexpress-reset reboot.15: Unable to restart (-22)
[ 36.161012] Reboot failed -- System halted
[ 56.140926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=2102 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 56.141055] Task dump for CPU 2:
[ 56.141096] mediaserver R running task 0 1832 1 0x00400000
[ 56.141170] Call trace:
[ 56.141234] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 119.190926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=8407 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 119.191055] Task dump for CPU 2:
[ 119.191096] mediaserver R running task 0 1832 1 0x00400000
[ 119.191171] Call trace:
[ 119.191235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 182.240926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=14712 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 182.241055] Task dump for CPU 2:
[ 182.241096] mediaserver R running task 0 1832 1 0x00400000
[ 182.241171] Call trace:
[ 182.241235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 245.290926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=21017 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 245.291056] Task dump for CPU 2:
[ 245.291097] mediaserver R running task 0 1832 1 0x00400000
[ 245.291171] Call trace:
[ 245.291236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 308.340926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=27322 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 308.341056] Task dump for CPU 2:
[ 308.341097] mediaserver R running task 0 1832 1 0x00400000
[ 308.341171] Call trace:
[ 308.341236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 371.390926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=33627 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 371.391055] Task dump for CPU 2:
[ 371.391096] mediaserver R running task 0 1832 1 0x00400000
[ 371.391171] Call trace:
[ 371.391235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 434.440926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=39932 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 434.441056] Task dump for CPU 2:
[ 434.441097] mediaserver R running task 0 1832 1 0x00400000
[ 434.441171] Call trace:
[ 434.441236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 497.490927] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=46237 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 497.491056] Task dump for CPU 2:
[ 497.491097] mediaserver R running task 0 1832 1 0x00400000
[ 497.491171] Call trace:
[ 497.491236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
<LAVA_DISPATCHER>2014-04-03 09:02:01 AM CRITICAL: The home screen was not displayed
[ 560.540926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=52542 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 560.541055] Task dump for CPU 2:
[ 560.541096] mediaserver R running task 0 1832 1 0x00400000
[ 560.541171] Call trace:
[ 560.541235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 623.590926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=58847 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 623.591055] Task dump for CPU 2:
[ 623.591097] mediaserver R running task 0 1832 1 0x00400000
[ 623.591171] Call trace:
[ 623.591235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
[ 686.640926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=65152 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 686.641056] Task dump for CPU 2:
[ 686.641097] mediaserver R running task 0 1832 1 0x00400000
[ 686.641171] Call trace:
[ 686.641236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
<LAVA_DISPATCHER>2014-04-03 09:12:02 AM WARNING: Sending newline in case of corruption.
[ 749.690926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=71457 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 749.691056] Task dump for CPU 2:
[ 749.691097] mediaserver R running task 0 1832 1 0x00400000
[ 749.691171] Call trace:
[ 749.691236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
<LAVA_DISPATCHER>2014-04-03 09:14:02 AM WARNING: Sending newline in case of corruption.
<LAVA_DISPATCHER>2014-04-03 09:16:02 AM WARNING: Sending newline in case of corruption.
[ 812.740926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=77762 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 812.741056] Task dump for CPU 2:
[ 812.741097] mediaserver R running task 0 1832 1 0x00400000
[ 812.741171] Call trace:
[ 812.741236] [<ffffffc00008498c>] __switch_to+0x74/0x8c
<LAVA_DISPATCHER>2014-04-03 09:18:02 AM WARNING: Sending newline in case of corruption.
[ 875.790926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=84067 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 875.791055] Task dump for CPU 2:
[ 875.791096] mediaserver R running task 0 1832 1 0x00400000
[ 875.791171] Call trace:
[ 875.791235] [<ffffffc00008498c>] __switch_to+0x74/0x8c
<LAVA_DISPATCHER>2014-04-03 09:20:02 AM WARNING: Sending newline in case of corruption.
<LAVA_DISPATCHER>2014-04-03 09:22:02 AM WARNING: Sending newline in case of corruption.
[ 938.840926] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=90372 jiffies, g=18446744073709551488, c=18446744073709551487, q=161)
[ 938.841055] Task dump for CPU 2:
[ 938.841097] mediaserver R running task 0 1832 1 0x00400000
[ 938.841171] Call trace:
[ 938.841236] [<ffffffc00008498c>] __switch_to+0x74/0x8c

Before it happened, there were some errors related to the low memory:

[ 14.425845] lowmemorykiller: lowmem_shrink: convert oom_adj to oom_score_adj:
[ 14.425900] lowmemorykiller: oom_adj 0 => oom_score_adj 0
[ 14.425952] lowmemorykiller: oom_adj 1 => oom_score_adj 58
[ 14.426004] lowmemorykiller: oom_adj 2 => oom_score_adj 117
[ 14.426056] lowmemorykiller: oom_adj 4 => oom_score_adj 235
[ 14.426108] lowmemorykiller: oom_adj 7 => oom_score_adj 411
[ 14.426161] lowmemorykiller: oom_adj 15 => oom_score_adj 1000

Also, I guess the failure of reboot to recovery mode is caused by:

[ 14.458237] init: cannot find '/system/etc/install-recovery.sh', disabling 'flash_recovery'

[Alex Shi (alex-shi)]

healthd is old issue. and IIRC, it is also a APP issue.

[Botao (botao-sun)]

The important thing is boot failed by this issue thus all tests associated got blocked.

[Botao (botao-sun)]

@Vishal, is this branch "linaro-android-armv8-lsk-pre-merge-ci" has been dropped and replaced by "linaro-android-armv8-lsk"? I ask because in comment #2 of Bug 1306489 you mentioned a new build on "armv8-linaro", which I assume it is this one:

https://validation.linaro.org/dashboard/image-reports/linaro-android-armv8-lsk

If so, then we may can remove that pre-merge one from dashboard.