[highbank] Oops: PC is at pl330_irq_handler+0x21c/0x3b0 [pl330]

[dann frazier]

SRU Justification:

    Impact: kernel panic on boot
    Fix: see attached patch.
    Testcase: boot the board and see if it panics or not.

===

Observed on a random boot (just once, so far):
[ 6.273265] Unable to handle kernel paging request at virtual address fffffffc
[ 6.280488] pgd = ec074000
[ 6.283187] [fffffffc] *pgd=2e7f5821, *pte=00000000, *ppte=00000000
[ 6.289458] Internal error: Oops: 17 [#1] SMP ARM
[ 6.294156] Modules linked in: pl330(F+) rtc_pl031(F)
[ 6.299233] CPU: 0 Tainted: GF W (3.8.0-17-generic #27-Ubuntu)
[ 6.306204] PC is at pl330_irq_handler+0x21c/0x3b0 [pl330]
[ 6.311699] LR is at l2x0_cache_sync+0x50/0x54
[ 6.316141] pc : [<bf0067a8>] lr : [<c0026940>] psr: 60000193
[ 6.316141] sp : edf07e98 ip : edf07e80 fp : edf07edc
[ 6.327616] r10: 00000001 r9 : 00000001 r8 : edc40480
[ 6.332840] r7 : 00000001 r6 : f00a0000 r5 : edc40480 r4 : 00000000
[ 6.339361] r3 : ffffffc8 r2 : fffffff9 r1 : 00000001 r0 : 00000000
[ 6.345884] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
[ 6.353101] Control: 10c5387d Table: 2c07404a DAC: 00000015
[ 6.358843] Process upstart-udev-br (pid: 318, stack limit = 0xedf06238)
[ 6.365540] Stack: (0xedf07e98 to 0xedf08000)
[ 6.369902] 7e80: c04d54e4 c00268fc
[ 6.378081] 7ea0: ffff6d21 a0000193 edc40484 edc40840 00000000 edd2d380 ed80be90 0000007c
[ 6.386259] 7ec0: 00000000 00000000 0000006f c0a5dae0 edf07f1c edf07ee0 c00ce5bc bf006598
[ 6.394432] 7ee0: 00000001 c007147c edf07f14 ed80be40 edf07f14 ed80be40 ed80be90 00000000
[ 6.402602] 7f00: f0002100 00000000 0000006f bece99c0 edf07f3c edf07f20 c00ce7ec c00ce568
[ 6.410772] 7f20: 00000000 ed80be40 ed80be90 00000000 edf07f54 edf07f40 c00d1a14 c00ce798
[ 6.418942] 7f40: 0000007c 0000007c edf07f6c edf07f58 c00cdee0 c00d1994 000000a0 c09b8690
[ 6.427112] 7f60: edf07f8c edf07f70 c00154d0 c00cdebc 00000010 f000210c c09c2524 edf07fb0
[ 6.435282] 7f80: edf07fac edf07f90 c0008578 c0015494 b6ea91c0 000f0030 ffffffff 00000007
[ 6.443451] 7fa0: 00000000 edf07fb0 c0654f00 c000854c bece98d8 bece98d4 00000001 00000009
[ 6.451621] 7fc0: bece9988 00000010 bece98e8 00000007 00000000 0000006f bece99c0 bece9908
[ 6.459791] 7fe0: 00000000 bece98c8 b6e9a37b b6ea91c0 000f0030 ffffffff 00000000 00000000
[ 6.467988] [<bf0067a8>] (pl330_irq_handler+0x21c/0x3b0 [pl330]) from [<c00ce5bc>] (handle_irq_event_percpu+0x60/0x230)
[ 6.478768] [<c00ce5bc>] (handle_irq_event_percpu+0x60/0x230) from [<c00ce7ec>] (handle_irq_event+0x60/0x80)
[ 6.488592] [<c00ce7ec>] (handle_irq_event+0x60/0x80) from [<c00d1a14>] (handle_fasteoi_irq+0x8c/0x164)
[ 6.497980] [<c00d1a14>] (handle_fasteoi_irq+0x8c/0x164) from [<c00cdee0>] (generic_handle_irq+0x30/0x40)
[ 6.507544] [<c00cdee0>] (generic_handle_irq+0x30/0x40) from [<c00154d0>] (handle_IRQ+0x48/0xa4)
[ 6.516325] [<c00154d0>] (handle_IRQ+0x48/0xa4) from [<c0008578>] (gic_handle_irq+0x38/0x6c)
[ 6.524764] [<c0008578>] (gic_handle_irq+0x38/0x6c) from [<c0654f00>] (__irq_usr+0x40/0x60)
[ 6.533104] Exception stack(0xedf07fb0 to 0xedf07ff8)
[ 6.538147] 7fa0: bece98d8 bece98d4 00000001 00000009
[ 6.546317] 7fc0: bece9988 00000010 bece98e8 00000007 00000000 0000006f bece99c0 bece9908
[ 6.554486] 7fe0: 00000000 bece98c8 b6e9a37b b6ea91c0 000f0030 ffffffff
[ 6.561097] Code: e3a01001 e5980098 e0622182 e0803182 (e593e034)
[ 6.567187] ---[ end trace da227214a82491ba ]---
[ 6.571796] Kernel panic - not syncing: Fatal exception in interrupt

[dann frazier (dannf)]

Actually, I'm now seeing this on every boot after power cycle (4 in a row). Only peculiar thing between these boots and previous ones is that I'd tried to test a kexec boot (which failed) - though I'm doing a BMC-initiated reset between every boot so I wouldn't think that'd be related.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1166956

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[dann frazier (dannf)]

Marking confirmed; system doesn't get far enough to run apport-collect (and it won't work behind my firewall anyhow)

[Chris Van Hoof (vanhoof)]

Built Raring master-next which pulls in rtc-pl031 as a built-in versus an external module. Though I'm not seeing a proper rtc sync at initial boot the following backtrace is evident:

Starting kernel ...

[ 0.000000] Booting Linux on physical CPU 0x900
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.8.0-17-generic (root@tangerine) (gcc version 4.7.3 (Ubuntu/Linaro 4.7.2-23ubuntu2) ) #28~lp1166956 SMP Tue Apr 9 21:06:15 UTC 2013 (Ubuntu 3.8.0-17.28~lp1166956-generic 3.8.6)
[ 0.000000] CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine: Highbank, model: Calxeda Highbank
[ 0.000000] cma: CMA: reserved 16 MiB at 2e800000
[ 0.000000] Memory policy: ECC disabled, Data cache writealloc
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] WARNING: at /home/vanhoof/scratch/ubuntu-raring/arch/arm/kernel/devtree.c:144 arm_dt_init_cpu_maps+0xfc/0x1ac()
[ 0.000000] DT /cpu 5 nodes greater than max cores 4, capping them
[ 0.000000] Modules linked in:
[ 0.000000] [<c001df98>] (unwind_backtrace+0x0/0x104) from [<c064a8b4>] (dump_stack+0x20/0x24)
[ 0.000000] [<c064a8b4>] (dump_stack+0x20/0x24) from [<c004b468>] (warn_slowpath_common+0x5c/0x74)
[ 0.000000] [<c004b468>] (warn_slowpath_common+0x5c/0x74) from [<c004b53c>] (warn_slowpath_fmt+0x40/0x48)
[ 0.000000] [<c004b53c>] (warn_slowpath_fmt+0x40/0x48) from [<c093ce78>] (arm_dt_init_cpu_maps+0xfc/0x1ac)
[ 0.000000] [<c093ce78>] (arm_dt_init_cpu_maps+0xfc/0x1ac) from [<c093bc40>] (setup_arch+0x578/0x754)
[ 0.000000] [<c093bc40>] (setup_arch+0x578/0x754) from [<c093761c>] (start_kernel+0xc0/0x360)
[ 0.000000] [<c093761c>] (start_kernel+0xc0/0x360) from [<0000807c>] (0x807c)
[ 0.000000] ---[ end trace da227214a82491b7 ]---
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] WARNING: at /home/vanhoof/scratch/ubuntu-raring/arch/arm/kernel/devtree.c:153 arm_dt_init_cpu_maps+0x12c/0x1ac()
[ 0.000000] DT missing boot CPU MPIDR[23:0], fall back to default cpu_logical_map
[ 0.000000] Modules linked in:
[ 0.000000] [<c001df98>] (unwind_backtrace+0x0/0x104) from [<c064a8b4>] (dump_stack+0x20/0x24)
[ 0.000000] [<c064a8b4>] (dump_stack+0x20/0x24) from [<c004b468>] (warn_slowpath_common+0x5c/0x74)
[ 0.000000] [<c004b468>] (warn_slowpath_common+0x5c/0x74) from [<c004b53c>] (warn_slowpath_fmt+0x40/0x48)
[ 0.000000] [<c004b53c>] (warn_slowpath_fmt+0x40/0x48) from [<c093cea8>] (arm_dt_init_cpu_maps+0x12c/0x1ac)
[ 0.000000] [<c093cea8>] (arm_dt_init_cpu_maps+0x12c/0x1ac) from [<c093bc40>] (setup_arch+0x578/0x754)
[ 0.000000] [<c093bc40>] (setup_arch+0x578/0x754) from [<c093761c>] (start_kernel+0xc0/0x360)
[ 0.000000] [<c093761c>] (start_kernel+0xc0/0x360) from [<0000807c>] (0x807c)
[ 0.000000] ---[ end trace da227214a82491b8 ]---

[Paolo Pisati (p-pisati)]

please give it a spin:

http://people.canonical.com/~ppisati/linux-image-3.8.0-18-generic_3.8.0-18.28~NOpl330_armhf.deb

it fixes it

[Paolo Pisati (p-pisati)]

some more info:

we actually never had the pl330 to work on 3.5:

http://paste.ubuntu.com/5701711/

it never really attached, and when it started to attach on 3.8 we found it buggy:

http://paste.ubuntu.com/5701455/

[Paolo Pisati (p-pisati)]

since the stack trace says we came from l2x0_cache_sync():

[ 6.191016] PC is at pl330_irq_handler+0x21c/0x3b0 [pl330]
[ 6.196512] LR is at l2x0_cache_sync+0x50/0x54

i wonder if it has anything to do with multiplatform and ERRATAs...

[Paolo Pisati (p-pisati)]

cache_sync() has nothing to do with any ERRATAs, i chasing ghosts :)

test the above kernel and report, please.

[dann frazier (dannf)]

that test kernel worked for me

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0001-UBUNTU-Config-disable-PL330_DMA.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Julian Wiedmann (jwiedmann)]

The patch from comment #11 was released with 3.8.0-19.29, but not applied to saucy.