kernel crash on arndale while CPU hotplug

[Naresh Kamboju]

pm-qa tests reported kernel crash on arndale

test case:
---------------------
test the cpu state is reflected in sysfs

name : cpuhotplug_02
script :
for each cpu
set cpuX offline
check cpuX status is offline
set cpuX back online
check cpuX status is online

Kernel crash dump:
----------------------------
###
### cpuhotplug_02:
### test the cpu state is reflected in sysfs
### https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/QA/Scripts#cpuhotplug_02
###

4IRQ153 no longer affine to CPU1

5CPU1: shutdown
CPU1: Booted secondary processor

0Kernepuhotplug_02.l/cpu1: checki g cpu is offlpne... a Ok
nic - not syncing: unexpected prefetch abort in Hyp mode at: 0x403f3040unexpected data abort in Hyp mode at: 0x803ec505unexpected HVC/SVC trap in Hyp mode at: 0x806c65e0

dCPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.0-1-linaro-arndale #1ubuntu1~ci+130602104315-Ubuntu
[<8001c385>] (unwind_backtrace+0x1/0x9c) from [<8001a625>] (show_stack+0x11/0x14)
[<8001a625>] (show_stack+0x11/0x14) from [<803ec565>] (panic+0x61/0x158)
[<803ec565>] (panic+0x61/0x158) from [<800118f7>] (cpu_init_hyp_mode+0x1b/0x5c)
[<800118f7>] (cpu_init_hyp_mode+0x1b/0x5c) from [<80011953>] (hyp_init_cpu_notify+0x1b/0x20)
[<80011953>] (hyp_init_cpu_notify+0x1b/0x20) from [<8004180d>] (notifier_call_chain+0x45/0x54)
[<8004180d>] (notifier_call_chain+0x45/0x54) from [<800297d9>] (__cpu_notify+0x1d/0x34)
[<800297d9>] (__cpu_notify+0x1d/0x34) from [<803ea455>] (secondary_start_kernel+0xa5/0xe4)
[<803ea455>] (secondary_start_kernel+0xa5/0xe4) from [<803e8755>] (exynos_cpu_die+0xa1/0xd4)

3drm_kms_helper: panic occurred, switching back to text console

2CPU0: stopping

dCPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-1-linaro-arndale #1ubuntu1~ci+130602104315-Ubuntu
[<8001c385>] (unwind_backtrace+0x1/0x9c) from [<8001a625>] (show_stack+0x11/0x14)
[<8001a625>] (show_stack+0x11/0x14) from [<8001b8d3>] (handle_IPI+0x10b/0x140)
[<8001b8d3>] (handle_IPI+0x10b/0x140) from [<800084cf>] (gic_handle_irq+0x53/0x58)
[<800084cf>] (gic_handle_irq+0x53/0x58) from [<8001741b>] (__irq_svc+0x3b/0x5c)
Exception stack(0x80659f18 to 0x80659f60)
9f00: 80659f60 3b9aca00
9f20: dab1c3e0 00000052 da9cbfab 00000052 8175b398 00000000 8072e8c4 806c4c84
9f40: 00000000 80658000 29aaaaab 80659f60 80342041 80331b70 600f0033 ffffffff
[<8001741b>] (__irq_svc+0x3b/0x5c) from [<80331b70>] (cpuidle_enter_state+0x38/0xa8)
[<80331b70>] (cpuidle_enter_state+0x38/0xa8) from [<80331c59>] (cpuidle_idle_call+0x79/0x140)
[<80331c59>] (cpuidle_idle_call+0x79/0x140) from [<800184ad>] (arch_cpu_idle+0xd/0x28)
[<800184ad>] (arch_cpu_idle+0xd/0x28) from [<80053159>] (cpu_startup_entry+0x5d/0x164)
[<80053159>] (cpu_startup_entry+0x5d/0x164) from [<80602639>] (start_kernel+0x291/0x2ec)

d------------[ cut here ]------------
2kernel BUG at /build/buildd/linux-linaro-arndale-3.10-3.10.0/kernel/sched/core.c:1213!

0Internal error: Oops - BUG: 0 [#1] SMP THUMB2

dModules linked in: gator

dCPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.0-1-linaro-arndale #1ubuntu1~ci+130602104315-Ubuntu

dtask: ec0d3a80 ti: ec0f0000 task.ti: ec0f0000
PC is at select_fallback_rq+0xf2/0x144
LR is at select_fallback_rq+0xbd/0x144
pc : [<803ecbee>] lr : [<803ecbb9>] psr: 600f01f3
sp : ec0f1d10 ip : 00000002 fp : 8067a984
r10: 00000000 r9 : 00000002 r8 : 803f6ad8
r7 : ec0f1d10 r6 : ec6eb740 r5 : ec6eb5c0 r4 : 8067a0f8
r3 : 00000000 r2 : 00000002 r1 : 00000002 r0 : 00000002
Flags: nZCv IRQs off FIQs off Mode SVC_32 ISA Thumb Segment kernel
Control: 70c5387d Table: 40003000 DAC: 55555555

Linux kernel:
 Linux version 3.10.0-1-linaro-arndale (buildd@alphard) (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) #1ubuntu1~ci+130602104315-Ubuntu SMP Sun Jun 2 13:40:03 UTC 2013

target:
Machine: SAMSUNG EXYNOS5 (Flattened Device Tree), model: Insignal Arndale evaluation board based on EXYNOS5250

LAVA test log:
http://validation.linaro.org/lava-server/scheduler/job/57740/log_file

build image:
http://snapshots.linaro.org/ubuntu/pre-built/arndale/361/arndale-raring_server_20130611-361.img.gz

[Amit Kucheria (amitk)]

Broken hotplug in the kernel can't be the fault of the QA testsuite. Passing to Samsung LT.

[Fathi Boudra (fboudra)]

Linaro U-boot and Linux Linaro are patched with respectively:
https://git.linaro.org/people/andrey.konovalov/u-boot.git/commit/689f7480db42d97553dcddffbacfea8ced4e49ea
https://git.linaro.org/landing-teams/working/samsung/kernel.git/commit/d0799d335287e87ffdcfbbbe6ef09a2c07a55ee0

The issue is triggered when U-Boot sets the CPU cores in HYP mode, but the kernel doesn't do so during hotplug (like mainline or LSK).

The issue isn't triggered with upstream U-Boot as the CPUs are booting into SVC mode.

Ideally, we should upstream the patch and backport to LSK. Opinions?

[Alex Shi (alex-shi)]

The patch is in discussion in lkml lkml.org/lkml/2013/5/7/135
we'd better wait for final solution for this bug. and then maybe final fix patch will be in LTS 3.10 too.

[Amit Kucheria (amitk)]

That seem like a very old patch (7th may 2013). Have there been updates?

On Fri, Jun 13, 2014 at 11:22 AM, Alex Shi <email address hidden> wrote:
> The patch is in discussion in lkml lkml.org/lkml/2013/5/7/135
> we'd better wait for final solution for this bug. and then maybe final fix patch will be in LTS 3.10 too.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1189839
>
> Title:
> kernel crash on arndale while CPU hotplug
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-landing-team-samsung/+bug/1189839/+subscriptions

[Fathi Boudra (fboudra)]

No, nobody followed up.