BUG: unable to handle kernel NULL pointer dereference at 0000000000000098 RIP: 0010:[<ffffffffc05818b0>] [<ffffffffc05818b0>] xfs_da3_node_read+0x30/0xd0 [xfs]

[concubidated]

DMESG Output:

[29864.180022] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
[29864.180599] IP: [<ffffffffc05818b0>] xfs_da3_node_read+0x30/0xd0 [xfs]
[29864.181004] PGD 1dec6cc067 PUD 1dedd37067 PMD 0
[29864.181298] Oops: 0000 [#1] SMP
[29864.181495] Modules linked in: xfs libcrc32c dm_crypt ses enclosure ipmi_devintf x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev sb_edac edac_core ipmi_si ipmi_msghandler mei_me ioatdma mei wmi lpc_ich mac_hid bonding 8021q garp stp mrp llc lp parport btrfs xor raid6_pq hid_generic ixgbe igb isci mpt2sas i2c_algo_bit usbhid dca ahci mdio ptp hid libsas raid_class libahci pps_core scsi_transport_sas
[29864.184685] CPU: 23 PID: 102617 Comm: ceph-osd Not tainted 3.16.0-38-generic #52~14.04.1-Ubuntu
[29864.185183] Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.0b 05/27/2014
[29864.185733] task: ffff883d993c9460 ti: ffff883bd319c000 task.ti: ffff883bd319c000
[29864.186168] RIP: 0010:[<ffffffffc05818b0>] [<ffffffffc05818b0>] xfs_da3_node_read+0x30/0xd0 [xfs]
[29864.186731] RSP: 0018:ffff883bd319fcf0 EFLAGS: 00010286
[29864.187039] RAX: 0000000000000000 RBX: ffff88359d81d960 RCX: 0000000000000001
[29864.187444] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff883bd319fca8
[29864.187841] RBP: ffff883bd319fd10 R08: 0000000000000001 R09: ffff883bd319fb90
[29864.188236] R10: ffff883bd319fc20 R11: 0000000000000001 R12: ffff883bd319fd40
[29864.188631] R13: 0000000000000002 R14: 0000000000000002 R15: 0000000056a70dc7
[29864.189046] FS: 00007f5cde432700(0000) GS:ffff88407fd60000(0000) knlGS:0000000000000000
[29864.209387] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[29864.219894] CR2: 0000000000000098 CR3: 0000001eedf12000 CR4: 00000000001407e0
[29864.230351] Stack:
[29864.240409] ffffffffc05c22b0 ffff883bd319fdc0 ffff883bd319fdc0 ffff8826033ff400
[29864.261066] ffff883bd319fd80 ffffffffc0544e7b 0000000200000008 ffff883d5579e010
[29864.282345] 000000013b75ec00 ffff8835c9534900 0000000000000000 0000000000000000
[29864.303671] Call Trace:
[29864.313888] [<ffffffffc0544e7b>] xfs_attr3_node_inactive+0x19b/0x220 [xfs]
[29864.324268] [<ffffffffc0544f96>] xfs_attr3_root_inactive+0x96/0x100 [xfs]
[29864.334448] [<ffffffffc05450b2>] xfs_attr_inactive+0xb2/0x150 [xfs]
[29864.344719] [<ffffffffc059628a>] xfs_inactive+0x8a/0x160 [xfs]
[29864.354423] [<ffffffffc055e3de>] xfs_fs_evict_inode+0x7e/0xc0 [xfs]
[29864.364804] [<ffffffff811ef204>] evict+0xb4/0x180
[29864.374653] [<ffffffff811ef9e5>] iput+0xf5/0x180
[29864.383760] [<ffffffff811e4413>] do_unlinkat+0x193/0x2c0
[29864.394371] [<ffffffff811d961f>] ? SYSC_newstat+0x2f/0x40
[29864.404994] [<ffffffff811e5496>] SyS_unlink+0x16/0x20
[29864.415451] [<ffffffff8176da4d>] system_call_fastpath+0x1a/0x1f
[29864.425331] Code: 55 48 89 e5 41 54 4d 89 c4 53 48 89 fb 48 83 ec 10 48 c7 04 24 b0 22 5c c0 e8 ed fe ff ff 85 c0 75 49 48 85 db 74 44 49 8b 34 24 <48> 8b 96 98 00 00 00 0f b7 52 08 66 c1 c2 08 66 81 fa be 3e 74
[29864.456197] RIP [<ffffffffc05818b0>] xfs_da3_node_read+0x30/0xd0 [xfs]
[29864.466910] RSP <ffff883bd319fcf0>
[29864.477302] CR2: 0000000000000098
[29864.502776] ---[ end trace 35d91394b621d272 ]---

Uname -a:

Linux <hostname> 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -rd:

Description: Ubuntu 14.04.1 LTS
Release: 14.04

This is occurring on multiple storage servers running Ceph, resulting in a system reboot and fsck for the filesystem. Has been occurring multiple times a day now.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alex Gorbachev (rwood-b)]

We were seeing this too on Ubuntu 14.04.2 and Ceph Firefly, after upgrade to kernel 4.1 we are now seeing:

Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261899] BUG: unable to handle kernel paging request at 000000190000001c
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261923] IP: [<ffffffff8118e476>] find_get_entries+0x66/0x160
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261941] PGD 1035954067 PUD 0
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261955] Oops: 0000 [#1] SMP
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.261969] Modules linked in: xfs libcrc32c ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich joydev mei_me mei ioatdma wmi 8021q ipmi_si garp 8250_fintek mrp ipmi_msghandler stp llc bonding mac_hid lp parport mlx4_en vxlan ip6_udp_tunnel udp_tunnel hid_generic usbhid hid igb ahci mpt2sas mlx4_core i2c_algo_bit libahci dca raid_class ptp scsi_transport_sas pps_core arcmsr
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262182] CPU: 10 PID: 8711 Comm: ceph-osd Not tainted 4.1.0-040100-generic #201506220235
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262197] Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262215] task: ffff8800721f1420 ti: ffff880fbad54000 task.ti: ffff880fbad54000
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262229] RIP: 0010:[<ffffffff8118e476>] [<ffffffff8118e476>] find_get_entries+0x66/0x160
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262248] RSP: 0018:ffff880fbad571a8 EFLAGS: 00010246
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262258] RAX: ffff880004000158 RBX: 000000000000000e RCX: 0000000000000000
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262303] RDX: ffff880004000158 RSI: ffff880fbad571c0 RDI: 0000001900000000
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262347] RBP: ffff880fbad57208 R08: 00000000000000c0 R09: 00000000000000ff
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262391] R10: 0000000000000000 R11: 0000000000000220 R12: 00000000000000b6
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262435] R13: ffff880fbad57268 R14: 000000000000000a R15: ffff880fbad572d8
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262479] FS: 00007f98cb0e0700(0000) GS:ffff88103f480000(0000) knlGS:0000000000000000
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262551] CR2: 000000190000001c CR3: 0000001034f0e000 CR4: 00000000000407e0
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262596] Stack:
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262618] ffff880fbad571f8 ffff880cf6076b30 ffff880bdde05da8 00000000000000e6
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262669] 0000000000000100 ffff880cf6076b28 00000000000000b5 ffff880fbad57258
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262721] ffff880fbad57258 ffff880fbad572d8 ffffffffffffffff ffff880cf6076b28
Jul 3 03:42:06 roc-4r-sca020 kernel: [554036.262772] Call Trace:
Jul 3 03:42:06 roc-4r-sca020 kernel: [...

[concubidated (tbrekke)]

Had a script that was dropping page caches every 10 minutes. Disabling that fixed this from occurring.