APM Mustang board kernel crashes when running Ceph

[Steve Capper]

Hi,
We've been running Ceph on 4 APM Mustang boards within Linaro, running stock Ubuntu Utopic (problem occurs with Trusty too).

Essentially when I place the network under heavy load (by writing data to a Ceph filesystem backed by 3xOSDs), I experience kernel crashes. I've harvested a few backtraces from the serial port and will paste them below as separate comments.

I think the problem lies in the xgene driver, but I don't know enough to resolve where this problem is happening.

Cheers,
--
Steve
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.7-0ubuntu8
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
DistroRelease: Ubuntu 14.10
IwConfig: Error: [Errno 2] No such file or directory
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:

ProcFB:

ProcKernelCmdLine: console=ttyS0,115200n8 ro
ProcVersionSignature: Ubuntu 3.16.0-25.33-generic 3.16.7
RfKill: Error: [Errno 2] No such file or directory
Tags: utopic utopic
Uname: Linux 3.16.0-25-generic aarch64
UnreportableReason: The report belongs to a package that is not installed.
UpgradeStatus: Upgraded to utopic on 2014-11-17 (10 days ago)
UserGroups: sudo
WifiSyslog:

_MarkForUpload: True

[Steve Capper (steve-capper)]

[ 1397.840891] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[ 1397.848945] pgd = ffffffc00007d000
[ 1397.852327] [00000008] *pgd=00000043ec009003, *pmd=00000043ec00a003, *pte=00e0000078090407
[ 1397.860582] Internal error: Oops: 96000045 [#1] SMP
[ 1397.865434] Modules linked in: xfs gpio_dwapb gpio_generic xgene_rng uio_pdrv_genirq uio btrfs xor raid6_pq libcrc32c ahci_xgene ahci_platform libahci xgene_enet
[ 1397.879949] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-39-generic #66-Ubuntu
[ 1397.887479] task: ffffffc3ec180b00 ti: ffffffc3ec188000 task.ti: ffffffc3ec188000
[ 1397.894926] PC is at tcp_ack+0x55c/0xd8c
[ 1397.898828] LR is at tcp_rcv_established+0x430/0x668
[ 1397.903765] pc : [<ffffffc000562918>] lr : [<ffffffc00056395c>] pstate: a0000145
[ 1397.911122] sp : ffffffc3ec18b7e0
[ 1397.914417] x29: ffffffc3ec18b7e0 x28: 0000000000000000
[ 1397.919720] x27: ffffffc3e2a18f20 x26: 0000000000000004
[ 1397.925023] x25: 0000000000000001 x24: ffffffc3912648e8
[ 1397.930326] x23: ffffffc3e2a18e00 x22: 00000000fe1bd8f2
[ 1397.935628] x21: 00000000fe1be442 x20: ffffffc0008e2000
[ 1397.940930] x19: 0000000000000402 x18: 000000007c8df90a
[ 1397.946231] x17: 0000007f9e4b4220 x16: ffffffc0001b9410
[ 1397.951534] x15: 003b9aca00000000 x14: 000b906a68000000
[ 1397.956836] x13: ffffffffab964494 x12: 0000000000000000
[ 1397.962136] x11: 0000000000000000 x10: 000000010001acb5
[ 1397.967439] x9 : ffffffc3e2a18e00 x8 : 0000000068000a0a
[ 1397.972741] x7 : 00000000fa2440ce x6 : 00000000fc085c8c
[ 1397.978043] x5 : 0000000000000200 x4 : 0000000000000000
[ 1397.983346] x3 : ffffffc391264910 x2 : 0000000000000001
[ 1397.988649] x1 : 0000000000000000 x0 : ffffffc3e2a18f20
[ 1397.993949]
[ 1397.995429] Process swapper/3 (pid: 0, stack limit = 0xffffffc3ec188058)
[ 1398.002094] Stack: (0xffffffc3ec18b7e0 to 0xffffffc3ec18c000)
[ 1398.007811] b7e0: ec18b8b0 ffffffc3 0056395c ffffffc0 e2a18e00 ffffffc3 59466524 ffffffc3
[ 1398.015945] b800: e7b1fb00 ffffffc3 e7b1fb28 ffffffc3 00000020 00000000 5946651c ffffffc3
[ 1398.024081] b820: 59466510 ffffffc3 59466524 ffffffc3 00000064 00000000 00951040 ffffffc0
[ 1398.032216] b840: 83d68a00 ffffffc3 df667800 ffffffc3 ec18b8b0 ffffffc3 00000000 ffffffff
[ 1398.040350] b860: 0001accf 00000001 0001accf 00000030 00000000 00000000 00000000 00000030
[ 1398.048485] b880: 00000000 00000000 759f4fe4 13a769ad 00000000 0000005d 0000000f 00000030
[ 1398.056619] b8a0: ec18b8b0 ffffffc3 fc00159c ffffffff ec18b920 ffffffc3 0056bde0 ffffffc0
[ 1398.064753] b8c0: e7b1fb00 ffffffc3 e2a18e00 ffffffc3 e3cca780 ffffffc3 00000000 00000000
[ 1398.072888] b8e0: 00000000 00000000 5946651c ffffffc3 59466510 ffffffc3 0056bbe0 ffffffc0
[ 1398.081022] b900: e7b1fb00 ffffffc3 00000020 00000000 594664c0 ffffffc3 009c2000 ffffffc0
[ 1398.089156] b920: ec18b9a0 ffffffc3 0056f76c ffffffc0 e7b1fb00 ffffffc3 0094de40 ffffffc0
[ 1398.097290] b940: ec188000 ffffffc3 009c2000 ffffffc0 e2a18e70 ffffffc3 e2a18e00 ffffffc3
[ 1398.105424] b960: 00000000 00000000 0000911a 00000000 009c2000 ffffffc0 009c2000 ffffffc0
[ 1398.113558] b980: e2a18e70 ffffffc3 00963000 ffffffc0 ec18b9e0 ffffffc3 00517658 ffffffc0
[ 139...

[Steve Capper (steve-capper)]

 [ 713.389733] Unable to handle kernel paging request at virtual address 700000011
[ 713.397023] pgd = ffffffc00007d000
[ 713.400406] [700000011] *pgd=0000000000000000
[ 713.404755] Internal error: Oops: 96000005 [#1] SMP
[ 713.409606] Modules linked in: xfs gpio_dwapb gpio_generic xgene_rng uio_pdrv_genirq uio btrfs xor raid6_pq libcrc32c ahci_xgene ahci_platform libahci xgene_enet
[ 713.424124] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.13.0-39-generic #66-Ubuntu
[ 713.431655] task: ffffffc3ec180b00 ti: ffffffc3ec188000 task.ti: ffffffc3ec188000
[ 713.439103] PC is at eth_type_trans+0x118/0x1a0
[ 713.443612] LR is at xgene_enet_process_ring+0xf0/0x390 [xgene_enet]
[ 713.449933] pc : [<ffffffc0005347dc>] lr : [<ffffffbffc00231c>] pstate: 20000145
[ 713.457289] sp : ffffffc3ec18bbe0
[ 713.460584] x29: ffffffc3ec18bbe0 x28: 00000000000005e6
[ 713.465886] x27: ffffffc3df5faa68 x26: ffffffc00096db50
[ 713.471188] x25: ffffffc3dfb22000 x24: ffffffc3df9bce18
[ 713.476491] x23: 0000000000000001 x22: 000000000000003f
[ 713.481794] x21: 000000000000008a x20: ffffffc3df629140
[ 713.487096] x19: ffffffc3df9bcc18 x18: 00000000fbed16fb
[ 713.492400] x17: 0000007f9aee118c x16: ffffffc0001b9c14
[ 713.497702] x15: 0000007f9af69590 x14: 54663e0000000000
[ 713.503004] x13: 0000000000000020 x12: 0000000000000000
[ 713.508307] x11: 0000000000300000 x10: ffffffc000952924
[ 713.513610] x9 : 00000000b43d107f x8 : 000000000000ffff
[ 713.518913] x7 : 000000000000ffff x6 : 0000000000000000
[ 713.524214] x5 : 000000000000ffff x4 : ffffffc3df9b0003
[ 713.529516] x3 : 0000000000000002 x2 : 00000000000005d8
[ 713.534819] x1 : 0000000000000400 x0 : 0000000700000011
[ 713.540122]
[ 713.541603] Process swapper/3 (pid: 0, stack limit = 0xffffffc3ec188058)
[ 713.548268] Stack: (0xffffffc3ec18bbe0 to 0xffffffc3ec18c000)
[ 713.553984] bbe0: ec18bbf0 ffffffc3 fc00231c ffffffbf ec18bc70 ffffffc3 fc0026b8 ffffffbf
[ 713.562120] bc00: df9bcc90 ffffffc3 00000040 00000000 00000040 00000000 fffa1100 ffffffc3
[ 713.570255] bc20: 008e2000 ffffffc0 fffa1110 ffffffc3 00000000 00000000 ec188000 ffffffc3
[ 713.578390] bc40: 00000000 00000000 00834848 ffffffc0 00952924 ffffffc0 fc003378 ffffffbf
[ 713.586525] bc60: dfb22000 ffffffc3 00567bd8 000001ff ec18bca0 ffffffc3 0051639c ffffffc0
[ 713.594659] bc80: df9bcc90 ffffffc3 00000040 00000000 0000012c 00000000 fff9fd70 ffffffc3
[ 713.602793] bca0: ec18bd20 ffffffc3 000aa188 ffffffc0 00000003 00000000 00000001 00000000
[ 713.610928] bcc0: 008e2098 ffffffc0 ec188000 ffffffc3 00000003 00000000 00960000 ffffffc0
[ 713.619063] bce0: 00000100 00000000 008e2000 ffffffc0 008d75c0 ffffffc0 00000003 00000000
[ 713.627198] bd00: 008e20b0 ffffffc0 00834848 ffffffc0 00960a36 ffffffc0 0000a17b 00000001
[ 713.635333] bd20: ec18bdb0 ffffffc3 000aa5f4 ffffffc0 ec188000 ffffffc3 009c2000 ffffffc0
[ 713.643468] bd40: ec188000 ffffffc3 0000005c 00000000 00000000 00000000 ec188000 ffffffc3
[ 713.651603] bd60: 0096090b ffffffc0 0007d000 00000040 007c54a0 ffffffc0 00612cd8 ffffffc0
[ 713.659738] bd80: ec010c00 ffffffc3 008ef2b0 ffffffc0 007bf5d8 ffffffc0 0000a17a 00000001
[ 713.667873] bda0: 00200040 00000...

[Steve Capper (steve-capper)]

[ 846.594302] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[ 846.602369] pgd = ffffffc00007d000
[ 846.605755] [00000008] *pgd=00000043ec009003, *pmd=00000043ec00a003, *pte=00e0000078090407
[ 846.614010] Internal error: Oops: 96000046 [#1] SMP
[ 846.618861] Modules linked in: ceph libceph libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache gpio_dwapb gpio_generic xgene_rng uio_pdrv_genirq uio ahci_xgene ahci_platform libahci xgene_enet
[ 846.636885] CPU: 3 PID: 161 Comm: kworker/3:1 Not tainted 3.13.0-39-generic #66-Ubuntu
[ 846.644774] Workqueue: ceph-msgr con_work [libceph]
[ 846.649635] task: ffffffc3df531600 ti: ffffffc3dfabc000 task.ti: ffffffc3dfabc000
[ 846.657082] PC is at tcp_ack+0x55c/0xd8c
[ 846.660982] LR is at tcp_ack+0x40c/0xd8c
[ 846.664882] pc : [<ffffffc000562918>] lr : [<ffffffc0005627c8>] pstate: a0000145
[ 846.672238] sp : ffffffc3dfabf340
[ 846.675533] x29: ffffffc3dfabf340 x28: 0000000000000000
[ 846.680835] x27: ffffffc3dfa72420 x26: 0000000000000004
[ 846.686138] x25: 0000000000000001 x24: ffffffc3df4b2ce8
[ 846.691440] x23: ffffffc3dfa72300 x22: 00000000103b8910
[ 846.696741] x21: 00000000103ba558 x20: ffffffc0008e2000
[ 846.702044] x19: 0000000000000402 x18: 000000000000000e
[ 846.707346] x17: 0000000000000007 x16: 0000000000000001
[ 846.712646] x15: ffffffbffc1e96d0 x14: 0000000000000000
[ 846.717949] x13: 0000000000000000 x12: 0000000000000000
[ 846.723252] x11: 0000000000000200 x10: 0000000000000000
[ 846.728552] x9 : ffffffc3dfa72300 x8 : 000000006c000a0a
[ 846.733854] x7 : 000000000e2696e8 x6 : 00000000fdeaf190
[ 846.739156] x5 : 0000000000000200 x4 : 0000000000000000
[ 846.744459] x3 : ffffffc3df4b2d10 x2 : 0000000000000001
[ 846.749762] x1 : 0000000000000000 x0 : ffffffc3dfa72420
[ 846.755062]
[ 846.756542] Process kworker/3:1 (pid: 161, stack limit = 0xffffffc3dfabc058)
[ 846.763554] Stack: (0xffffffc3dfabf340 to 0xffffffc3dfac0000)
[ 846.769270] f340: dfabf410 ffffffc3 0056395c ffffffc0 dfa72300 ffffffc3 33798064 ffffffc3
[ 846.777404] f360: e552fc00 ffffffc3 e552fc28 ffffffc3 00000020 00000000 3379805c ffffffc3
[ 846.785539] f380: 33798050 ffffffc3 33798064 ffffffc3 00000064 00000000 00951040 ffffffc0
[ 846.793673] f3a0: 00235250 00000000 0095cd40 ffffffc0 00000140 00000000 00000000 ffffffff
[ 846.801807] f3c0: 0000d583 00000001 0000d583 00000038 00000000 00000000 00000000 00000038
[ 846.809943] f3e0: 00000000 00000000 789e02f0 13a80dff 00000000 00000038 0000000f 00000038
[ 846.818077] f400: dfabf550 ffffffc3 00156118 ffffffff dfabf480 ffffffc3 0056bde0 ffffffc0
[ 846.826211] f420: e552fc00 ffffffc3 dfa72300 ffffffc3 df9cfcc0 ffffffc3 00000000 00000000
[ 846.834345] f440: 00000000 00000000 3379805c ffffffc3 33798050 ffffffc3 0056bbe0 ffffffc0
[ 846.842479] f460: e552fc00 ffffffc3 00000020 00000000 33798000 ffffffc3 009c2000 ffffffc0
[ 846.850613] f480: dfabf500 ffffffc3 0056f76c ffffffc0 e552fc00 ffffffc3 0094de40 ffffffc0
[ 846.858747] f4a0: dfabc000 ffffffc3 009c2000 ffffffc0 dfa72370 ffffffc3 dfa72300 ffffffc3
[ 846.866881] f4c0: 00000000 00000000 0000901a 00000000 009c2000 ffffffc0 009c2000 ffffffc0
[ 846....

[Steve Capper (steve-capper)]

[ 2672.127929] Unable to handle kernel NULL pointer dereference at virtual address 00000008
[ 2672.136000] pgd = ffffffc00007d000
[ 2672.139383] [00000008] *pgd=00000043ec009003, *pmd=00000043ec00a003, *pte=00e0000078090407
[ 2672.147644] Internal error: Oops: 96000045 [#1] SMP
[ 2672.152496] Modules linked in: ceph libceph libcrc32c nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache gpio_dwapb gpio_generic xgene_rng uio_pdrv_genirq uio ahci_xgene ahci_platform libahci xgene_enet
[ 2672.170522] CPU: 3 PID: 167 Comm: kworker/3:1 Not tainted 3.13.0-39-generic #66-Ubuntu
[ 2672.178410] Workqueue: ceph-msgr con_work [libceph]
[ 2672.183270] task: ffffffc3e547a100 ti: ffffffc3e5678000 task.ti: ffffffc3e5678000
[ 2672.190717] PC is at tcp_ack+0x55c/0xd8c
[ 2672.194618] LR is at tcp_rcv_established+0x430/0x668
[ 2672.199555] pc : [<ffffffc000562918>] lr : [<ffffffc00056395c>] pstate: a0000145
[ 2672.206911] sp : ffffffc3e567b670
[ 2672.210206] x29: ffffffc3e567b670 x28: 0000000000000000
[ 2672.215509] x27: ffffffc3e7571d20 x26: 0000000000000004
[ 2672.220812] x25: 0000000000000001 x24: ffffffc37b9372e8
[ 2672.226114] x23: ffffffc3e7571c00 x22: 00000000a59a782f
[ 2672.231417] x21: 00000000a59a837f x20: ffffffc0008e2000
[ 2672.236719] x19: 0000000000000402 x18: 00000000a0313905
[ 2672.242021] x17: 0000007facf86220 x16: ffffffc0001b9410
[ 2672.247323] x15: ffffffbffc1e96d0 x14: 0000000000000000
[ 2672.252623] x13: 0000000000000000 x12: 0000000000000000
[ 2672.257926] x11: 0000000000300000 x10: ffffffc000952924
[ 2672.263229] x9 : 00000000a5ca782f x8 : 0000000000020f01
[ 2672.268532] x7 : 00000000a2e82fad x6 : 00000000fd4dac2e
[ 2672.273834] x5 : 0000000000000200 x4 : 0000000000000000
[ 2672.279137] x3 : ffffffc37b937310 x2 : 0000000000000001
[ 2672.284440] x1 : 0000000000000000 x0 : ffffffc3e7571d20
[ 2672.289742]
[ 2672.291224] Process kworker/3:1 (pid: 167, stack limit = 0xffffffc3e5678058)
[ 2672.298235] Stack: (0xffffffc3e567b670 to 0xffffffc3e567c000)
[ 2672.303951] b660: e567b740 ffffffc3 0056395c ffffffc0
[ 2672.312086] b680: e7571c00 ffffffc3 6e9f17a4 ffffffc3 e55bbd00 ffffffc3 e55bbd28 ffffffc3
[ 2672.320220] b6a0: 00000020 00000000 6e9f179c ffffffc3 6e9f1790 ffffffc3 6e9f17a4 ffffffc3
[ 2672.328355] b6c0: 00000064 00000000 e7571c00 ffffffc3 e567b710 ffffffc3 005669b4 ffffffc0
[ 2672.336490] b6e0: e7571c00 ffffffc3 00000000 ffffffff 00039e9d 00000001 00039e9d 00000033
[ 2672.344625] b700: 00000000 00000000 00000000 00000033 00000000 00000000 300e4f33 13a816f3
[ 2672.352759] b720: 00000000 0000003d 0000000f 00000033 00000020 00000000 00000000 ffffffff
[ 2672.360893] b740: e567b7b0 ffffffc3 0056bde0 ffffffc0 e55bbd00 ffffffc3 e7571c00 ffffffc3
[ 2672.369027] b760: df7ba900 ffffffc3 00000000 00000000 00000000 00000000 6e9f179c ffffffc3
[ 2672.377161] b780: 6e9f1790 ffffffc3 0056bbe0 ffffffc0 e55bbd00 ffffffc3 00000020 00000000
[ 2672.385295] b7a0: 6e9f1740 ffffffc3 009e2000 ffffffc0 e567b820 ffffffc3 00501368 ffffffc0
[ 2672.393429] b7c0: e55bbd00 ffffffc3 00000000 00000000 e7571c00 ffffffc3 009e2000 ffffffc0
[ 2672.401564] b7e0: 00960a34 ffffffc0 e7571c70 ffffffc3 00000001 00000000 00834140 ffff...

[Steve Capper (steve-capper)]

[11072.912028] skbuff: skb_over_panic: text:ffffffbffc002578 len:2994 put:1514 head:ffffffc17a810000 data:ffffffc17a810064 tail:0xc16 end:0x680 dev:<NULL>
[11072.925544] Kernel panic - not syncing: BUG!
[11072.929792] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.16.0-24-generic #32-Ubuntu
[11072.937321] Call trace:
[11072.939759] [<ffffffc0000899f4>] dump_backtrace+0x0/0x174
[11072.945129] [<ffffffc000089b84>] show_stack+0x1c/0x28
[11072.950154] [<ffffffc0007316e8>] dump_stack+0x74/0x94
[11072.955178] [<ffffffc00072f05c>] panic+0xe8/0x228
[11072.959857] [<ffffffc000623e20>] skb_panic+0x94/0xa4
[11072.964794] [<ffffffc0006264c4>] skb_put+0xa8/0xac
[11072.969565] [<ffffffbffc002574>] xgene_enet_process_ring+0xdc/0x38c [xgene_enet]
[11072.976922] [<ffffffbffc00292c>] xgene_enet_napi+0x28/0x5c [xgene_enet]
[11072.983503] [<ffffffc000636e68>] net_rx_action+0x1a4/0x238
[11072.988959] [<ffffffc0000b2db8>] __do_softirq+0x124/0x284
[11072.994329] [<ffffffc0000b31a8>] irq_exit+0x94/0xc8
[11072.999181] [<ffffffc0000858b0>] handle_IRQ+0x54/0xd0
[11073.004204] [<ffffffc0000812f0>] gic_handle_irq+0x44/0x88
[11073.009574] Exception stack(0xffffffc3f01dfe00 to 0xffffffc3f01dff20)
[11073.015981] fe00: f01dc000 ffffffc3 00b7f308 ffffffc0 f01dff40 ffffffc3 00086290 ffffffc0
[11073.024116] fe20: 000f26f4 ffffffc0 00000000 00000000 00c26f80 ffffffc0 00ae5260 ffffffc0
[11073.032251] fe40: 00ae4000 ffffffc0 00106fc3 00000001 25c17d03 00000002 0a24a0c1 00000000
[11073.040385] fe60: 00000018 00000000 f01dfd60 ffffffc3 00000001 00000000 000a13fc 00000000
[11073.048520] fe80: 00000003 00000000 00000008 00000000 00000000 00000000 ab365590 0000007f
[11073.056654] fea0: 00205ddc ffffffc0 ab2df664 0000007f 5a8dc000 00000055 f01dc000 ffffffc3
[11073.064789] fec0: 00b7f308 ffffffc0 00b83fa0 ffffffc0 00b63000 ffffffc0 00000000 00000000
[11073.072923] fee0: 00000000 00000000 0007b000 00000040 00abe000 ffffffc0 007420d0 ffffffc0
[11073.081058] ff00: ffffffed 00000000 f01dff40 ffffffc3 0008628c ffffffc0 f01dff40 ffffffc3
[11073.089192] [<ffffffc000084da0>] el1_irq+0x60/0xc0
[11073.093959] [<ffffffc0000f26f0>] cpu_startup_entry+0x258/0x2f0
[11073.099761] [<ffffffc000090d64>] secondary_start_kernel+0x10c/0x11c
[11073.105997] CPU3: stopping
[11073.108693] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.16.0-24-generic #32-Ubuntu
[11073.116223] Call trace:
[11073.118659] [<ffffffc0000899f4>] dump_backtrace+0x0/0x174
[11073.124029] [<ffffffc000089b84>] show_stack+0x1c/0x28
[11073.129053] [<ffffffc0007316e8>] dump_stack+0x74/0x94
[11073.134077] [<ffffffc000091318>] handle_IPI+0x194/0x1ac
[11073.139274] [<ffffffc00008132c>] gic_handle_irq+0x80/0x88
[11073.144644] Exception stack(0xffffffc3f01efe00 to 0xffffffc3f01eff20)
[11073.151051] fe00: f01ec000 ffffffc3 00b7f308 ffffffc0 f01eff40 ffffffc3 00086290 ffffffc0
[11073.159186] fe20: 000f26f4 ffffffc0 00000000 00000000 00c26f80 ffffffc0 00ae5260 ffffffc0
[11073.167321] fe40: 00ae4000 ffffffc0 00106fd7 00000001 e81c7918 ffffffc3 0a24a0c1 00000000
[11073.175455] fe60: 00000018 00000000 f01efd60 ffffffc3 00000400 00000000 000003ff 00000000
[11073.183590] fe80: 00000001 00000000 00000000 ffffffff 00000000 00000000 4c7ab300 ffffffc0
[11073.191724]...

[Steve Capper (steve-capper)]

[ 8943.714025] Unable to handle kernel NULL pointer dereference at virtual address 000000d8
[ 8943.722099] pgd = ffffffc0ffb4b000
[ 8943.725482] [000000d8] *pgd=0000000000000000
[ 8943.729747] Internal error: Oops: 96000005 [#1] SMP
[ 8943.734598] Modules linked in: xfs libcrc32c xgene_rng uio_pdrv_genirq uio btrfs xor raid6_pq realtek ahci_xgene libahci_platform libahci xgene_enet
[ 8943.747982] CPU: 3 PID: 1664 Comm: ceph-osd Not tainted 3.16.0-24-generic #32-Ubuntu
[ 8943.755685] task: ffffffc0fdf83d40 ti: ffffffc3e7208000 task.ti: ffffffc3e7208000
[ 8943.763133] PC is at xgene_enet_process_ring+0xc0/0x38c [xgene_enet]
[ 8943.769453] LR is at xgene_enet_process_ring+0xa4/0x38c [xgene_enet]
[ 8943.775774] pc : [<ffffffbffc002558>] lr : [<ffffffbffc00253c>] pstate: 80000145
[ 8943.783129] sp : ffffffc3e720b890
[ 8943.786425] x29: ffffffc3e720b890 x28: ffffffc3eb917a18
[ 8943.791727] x27: 0000000000000000 x26: 0000000000000000
[ 8943.797029] x25: ffffffc3df8cc018 x24: ffffffc000b7e518
[ 8943.802331] x23: ffffffc3eb090000 x22: 0000000000000001
[ 8943.807633] x21: 000000000000003f x20: ffffffc3df8aba00
[ 8943.812934] x19: 00000000000001d0 x18: 0000007f81ffd710
[ 8943.818238] x17: 0000007fa4d1ce34 x16: ffffffc0001f6b34
[ 8943.823539] x15: 0000007fa4da7590 x14: 0000000000000000
[ 8943.828841] x13: ffffffff00000000 x12: 0000000000000003
[ 8943.834142] x11: 00000000ffffffc0 x10: ffffffc3e65af000
[ 8943.839444] x9 : ffffffc0003dc1a4 x8 : 0000000000000003
[ 8943.844746] x7 : 0000000000000064 x6 : 00000040ffc00000
[ 8943.850049] x5 : ffffffc375aad540 x4 : 0000004100000000
[ 8943.855351] x3 : 00020000000001c4 x2 : 00000000000005ea
[ 8943.860654] x1 : 00000043e5b670ee x0 : ffffffc3eb9ae818
[ 8943.865955]
[ 8943.867437] Process ceph-osd (pid: 1664, stack limit = 0xffffffc3e7208058)
[ 8943.874274] Stack: (0xffffffc3e720b890 to 0xffffffc3e720c000)
[ 8943.879990] b880: e720b900 ffffffc3 fc002930 ffffffbf
[ 8943.888126] b8a0: eb917a90 ffffffc3 fff82d80 ffffffc3 00000040 00000000 00000040 00000000
[ 8943.896261] b8c0: 000d3024 00000001 fff82d90 ffffffc3 00ac6000 ffffffc0 eb917a90 ffffffc3
[ 8943.904396] b8e0: 00000000 00000000 009d3000 ffffffc0 eb090000 ffffffc3 00000000 000001ff
[ 8943.912529] b900: e720b930 ffffffc3 00636e6c ffffffc0 00000000 00000000 fff82d80 ffffffc3
[ 8943.920664] b920: 0000012c 00000000 e7208000 ffffffc3 e720b9b0 ffffffc3 000b2dbc ffffffc0
[ 8943.928799] b940: 00000003 00000000 00000003 00000000 00000004 00000000 00ac60d8 ffffffc0
[ 8943.936933] b960: 00000008 00000000 e7208000 ffffffc3 00000100 00000000 00ac6000 ffffffc0
[ 8943.945068] b980: 00abbb28 ffffffc0 e7208000 ffffffc3 000b2d50 ffffffc0 009d3bc8 ffffffc0
[ 8943.953203] b9a0: 00b6210b ffffffc0 e7208000 ffffffc3 e720ba30 ffffffc3 000b31ac ffffffc0
[ 8943.961338] b9c0: e7208000 ffffffc3 0000005c 00000000 00000000 00000000 00002010 ffffff80
[ 8943.969472] b9e0: 20000145 00000000 00000015 00000000 00000116 00000000 0000004f 00000000
[ 8943.977607] ba00: 00ad0000 ffffffc0 e7208000 ffffffc3 000d3023 00000001 00940aa8 ffffffc0
[ 8943.985742] ba20: 007423b8 ffffffc0 00400040 0000000a e720ba50 ffffffc3 000858b4 ffffffc0
[ 8943.993877] b...

[Steve Capper (steve-capper)]

  385.828557] Unable to handle kernel NULL pointer dereference at virtual address 000000d8
[ 385.836623] pgd = ffffffc0ff821000
[ 385.840006] [000000d8] *pgd=0000000000000000
[ 385.844271] Internal error: Oops: 96000005 [#1] SMP
[ 385.849123] Modules linked in: xfs libcrc32c xgene_rng uio_pdrv_genirq uio btrfs xor raid6_pq realtek ahci_xgene libahci_platform libahci xgene_enet
[ 385.862506] CPU: 6 PID: 1534 Comm: ceph-osd Not tainted 3.16.0-24-generic #32-Ubuntu
[ 385.870208] task: ffffffc3e955b100 ti: ffffffc0ff3c8000 task.ti: ffffffc0ff3c8000
[ 385.877656] PC is at xgene_enet_process_ring+0xc0/0x38c [xgene_enet]
[ 385.883976] LR is at xgene_enet_process_ring+0xa4/0x38c [xgene_enet]
[ 385.890295] pc : [<ffffffbffc002558>] lr : [<ffffffbffc00253c>] pstate: 80000145
[ 385.897652] sp : ffffffc0ff3cb460
[ 385.900946] x29: ffffffc0ff3cb460 x28: ffffffc3eb2cf418
[ 385.906248] x27: 0000000000000000 x26: 0000000000000000
[ 385.911551] x25: ffffffc3eb2cf618 x24: ffffffc000b7e518
[ 385.916853] x23: ffffffc3eb2eb000 x22: 0000000000000001
[ 385.922154] x21: 000000000000003f x20: ffffffc3df4b9940
[ 385.927457] x19: 00000000000000ca x18: 000000558ea41000
[ 385.932760] x17: 0000007f79635664 x16: ffffffc000205ddc
[ 385.938063] x15: 0000007f796bb590 x14: 0000000000000000
[ 385.943365] x13: 0000000000000010 x12: 0000000000000003
[ 385.948666] x11: 00000000ffffffc0 x10: ffffffc3ea783800
[ 385.953969] x9 : ffffffc0003dc1a4 x8 : 0000000021f52420
[ 385.959271] x7 : 0000100800000000 x6 : 00000040ffc00000
[ 385.964572] x5 : ffffffc3eb37a9f0 x4 : 0000004100000000
[ 385.969873] x3 : 0002000000000185 x2 : 0000000000000042
[ 385.975176] x1 : 00000043ea4840ee x0 : ffffffc3eb2f3018
[ 385.980479]
[ 385.981960] Process ceph-osd (pid: 1534, stack limit = 0xffffffc0ff3c8058)
[ 385.988798] Stack: (0xffffffc0ff3cb460 to 0xffffffc0ff3cc000)
[ 385.994515] b460: ff3cb4d0 ffffffc0 fc002930 ffffffbf eb2cf490 ffffffc3 fffa6d80 ffffffc3
[ 386.002650] b480: 00000040 00000000 00000040 00000000 00002185 00000001 fffa6d90 ffffffc3
[ 386.010784] b4a0: 00ac6000 ffffffc0 eb2cf490 ffffffc3 00000000 00000000 009d3000 ffffffc0
[ 386.018918] b4c0: eb2eb000 ffffffc3 00000010 000001ff ff3cb500 ffffffc0 00636e6c ffffffc0
[ 386.027052] b4e0: 00000000 00000000 fffa6d80 ffffffc3 0000012c 00000000 ff3c8000 ffffffc0
[ 386.035187] b500: ff3cb580 ffffffc0 000b2dbc ffffffc0 00000003 00000000 00000003 00000000
[ 386.043321] b520: 00000004 00000000 00ac60d8 ffffffc0 00000008 00000000 ff3c8000 ffffffc0
[ 386.051455] b540: 00000100 00000000 00ac6000 ffffffc0 00abbb28 ffffffc0 ff3cbae4 ffffffc0
[ 386.059589] b560: 000b2d50 ffffffc0 009d3bc8 ffffffc0 00b6210b ffffffc0 ff3c8000 ffffffc0
[ 386.067723] b580: ff3cb600 ffffffc0 000b31ac ffffffc0 ff3c8000 ffffffc0 0000005c 00000000
[ 386.075858] b5a0: 00000000 00000000 00002010 ffffff80 20000145 00000000 00000020 00000000
[ 386.083992] b5c0: df6e2c01 ffffffc3 ff3cb918 ffffffc0 df6e2c00 ffffffc3 ff3cbae4 ffffffc0
[ 386.092126] b5e0: 00002184 00000001 00940aa8 ffffffc0 007423b8 ffffffc0 00400040 0000000a
[ 386.100260] b600: ff3cb620 ffffffc0 000858b4 ffffffc0 00abd000 ffffffc0 000e6f6c ffffffc0
[ 386.108395] b6...

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1396979

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Steve Capper (steve-capper)]

apport information

[Paolo Pisati (p-pisati)]

can you try this kernel and report if it solves your problem?

http://people.canonical.com/~ppisati/linux-image-3.16.0-27-generic_3.16.0-27.36~b9651a5_arm64.deb

[Steve Capper (steve-capper)]

Thanks Paolo,
I've installed this kernel on our boards and we will be running stability tests on them.

Cheers,
--
Steve

[Steve Capper (steve-capper)]

Hi,

We get the following crash with the proposed kernel:

Cheers,
--
Steve

[251283.329169] Unable to handle kernel NULL pointer dereference at virtual address 000000d8
[251283.337318] pgd = ffffffc00007d000
[251283.340794] [000000d8] *pgd=00000043f0009003, *pmd=00000043f000a003, *pte=0060000078090707
[251283.349136] Internal error: Oops: 96000005 [#1] SMP
[251283.354074] Modules linked in: rbd libceph libcrc32c xgene_rng uio_pdrv_genirq uio btrfs xor raid6_pq realtek ahci_xgene libahci_platform libahci xgene_enet
[251283.368246] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.16.0-27-generic #36
[251283.375257] task: ffffffc3f01e0c40 ti: ffffffc3f01e8000 task.ti: ffffffc3f01e8000
[251283.382792] PC is at xgene_enet_process_ring+0xbc/0x384 [xgene_enet]
[251283.389199] LR is at xgene_enet_process_ring+0xa0/0x384 [xgene_enet]
[251283.395605] pc : [<ffffffbffc003794>] lr : [<ffffffbffc003778>] pstate: 80000145
[251283.403047] sp : ffffffc3f01ebbf0
[251283.406428] x29: ffffffc3f01ebbf0 x28: ffffffc3eb389a18
[251283.411816] x27: 0000000000000000 x26: 0000000000000000
[251283.417205] x25: ffffffc3eb326000 x24: ffffffc3eb389c18
[251283.422594] x23: ffffffc000b82518 x22: 00000000000001ff
[251283.427981] x21: 0000000000000001 x20: ffffffc3dfbadc80
[251283.433369] x19: 00000000000000e4 x18: 000000000000000d
[251283.438757] x17: 0000007f96714650 x16: ffffffc00011e00c
[251283.444145] x15: 0026996c58fec7d8 x14: 0000000000000000
[251283.449534] x13: 0000000000000001 x12: ffffffc0ff215c20
[251283.454922] x11: 0000000000000000 x10: 0000000020a049bb
[251283.460311] x9 : ffffffc0003dd3c8 x8 : 0000000000000000
[251283.465700] x7 : 0000000000000000 x6 : 00000040ffc00000
[251283.471088] x5 : 0000000000000000 x4 : 0000004100000000
[251283.476477] x3 : 00020000000000be x2 : 00000000000005ea
[251283.481866] x1 : 00000043dc7f80ee x0 : ffffffc3dfada018
[251283.487254]
[251283.488821] Process swapper/2 (pid: 0, stack limit = 0xffffffc3f01e8058)
[251283.495572] Stack: (0xffffffc3f01ebbf0 to 0xffffffc3f01ec000)
[251283.501375] bbe0: f01ebc60 ffffffc3 fc003b88 ffffffbf
[251283.509596] bc00: eb389a90 ffffffc3 fff75dc0 ffffffc3 00000040 00000000 00000040 00000000
[251283.517817] bc20: 017eef06 00000001 fff75dd0 ffffffc3 00aca000 ffffffc0 eb389a90 ffffffc3
[251283.526038] bc40: 00000000 00000000 009d4000 ffffffc0 eb326000 ffffffc3 e8000000 00000040
[251283.534258] bc60: f01ebc90 ffffffc3 00638018 ffffffc0 00000000 00000000 fff75dc0 ffffffc3
[251283.542479] bc80: 0000012c 00000000 f01e8000 ffffffc3 f01ebd10 ffffffc3 000b2e24 ffffffc0
[251283.550699] bca0: 00000003 00000000 00000003 00000000 00000004 00000000 00aca0d8 ffffffc0
[251283.558920] bcc0: 00000008 00000000 f01e8000 ffffffc3 00000100 00000000 00aca000 ffffffc0
[251283.567140] bce0: 00abcb28 ffffffc0 ffffffed 00000000 000b2db8 ffffffc0 009d4d50 ffffffc0
[251283.575361] bd00: 00b66146 ffffffc0 f01e8000 ffffffc3 f01ebd90 ffffffc3 000b3214 ffffffc0
[251283.583582] bd20: f01e8000 ffffffc3 0000005c 00000000 00000000 00000000 00002010 ffffff80
[251283.591802] bd40: 60000145 00000000 00000000 00000000 0007b000 00000040 00abf000 ffffffc0
[251283.600023] bd60: 007430d0 ffffffc0 f...