XFS Oopsing 4.2.0-35-generic Kernel - Block out of range

[John McEleney]

After updating the Kernel on two of our Trusty Ceph servers from 3.19.0-33-generic (vivid) to 4.2.0-35-generic (wily), both servers have now Oopsed multiple times accessing some of their XFS filesystems.

Once this error occurs, the XFS filesystem becomes permanently inaccessible. Even after a reboot, any attempt to access the filesystem in question either by mounting it, or attempting an xfs_repair will trigger the same Oops again.

These filesystems did not immediately fail until several hours after the upgrade. I suspect that this is related to some housekeeping being triggered within the filesystem.

This is the call trace from the moment a filesystem first failed:
Apr 28 22:47:25 ceph-store5 kernel: [26692.804773] ------------[ cut here ]------------
Apr 28 22:47:25 ceph-store5 kernel: [26692.810046] WARNING: CPU: 8 PID: 5195 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
Apr 28 22:47:25 ceph-store5 kernel: [26692.824715] Modules linked in: bridge openvswitch xfs dcdbas ipmi_devintf intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev dm_multipath sb_edac scsi_dh edac_core input_leds mei_me lpc_ich mei ipmi_si ipmi_msghandler 8250_fintek wmi acpi_power_meter 8021q acpi_pad mac_hid garp mrp stp llc shpchp lp parport btrfs bcache raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic xor usbhid hid raid6_pq bnx2x raid1 ptp mpt2sas raid0 pps_core multipath raid_class mdio scsi_transport_sas nvme libcrc32c linear
Apr 28 22:47:25 ceph-store5 kernel: [26692.891939] CPU: 8 PID: 5195 Comm: ceph-osd Tainted: G D W 4.2.0-35-generic #40~14.04.1-Ubuntu
Apr 28 22:47:25 ceph-store5 kernel: [26692.902689] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
Apr 28 22:47:25 ceph-store5 kernel: [26692.911303] 0000000000000000 ffff8800c7bf74a8 ffffffff817bcbf8 0000000000000000
Apr 28 22:47:25 ceph-store5 kernel: [26692.919677] ffffffffc0a7f300 ffff8800c7bf74e8 ffffffff81079b5a ffff8800c7bf7508
Apr 28 22:47:25 ceph-store5 kernel: [26692.928033] ffff88100a28ed80 0000000000000008 0000000000000000 ffff8800c7bf75f8
Apr 28 22:47:25 ceph-store5 kernel: [26692.936441] Call Trace:
Apr 28 22:47:25 ceph-store5 kernel: [26692.939208] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
Apr 28 22:47:25 ceph-store5 kernel: [26692.944992] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
Apr 28 22:47:25 ceph-store5 kernel: [26692.951752] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
Apr 28 22:47:25 ceph-store5 kernel: [26692.958346] [<ffffffffc0a3e12e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.965221] [<ffffffffc0a3e1aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.972194] [<ffffffffc0a0130a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.979763] [<ffffffffc0a69f26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.987412] [<ffffffffc0a17380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26692.994564] [<ffffffffc0a01d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.002302] [<ffffffffc0a3dd3f>] ? xfs_buf_trylock+0x1f/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.009379] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.016229] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
Apr 28 22:47:25 ceph-store5 kernel: [26693.022814] [<ffffffffc0a3281a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.029669] [<ffffffffc0a020a2>] xfs_alloc_vextent+0x1d2/0x660 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.036915] [<ffffffffc0a12746>] xfs_bmap_btalloc+0x376/0x6b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.044074] [<ffffffffc0a12aa4>] xfs_bmap_alloc+0x24/0x40 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.050833] [<ffffffffc0a13456>] xfs_bmapi_write+0x486/0x9a0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.057882] [<ffffffffc0a1d1ce>] xfs_da_grow_inode_int+0xae/0x2d0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.065403] [<ffffffff811d1bc2>] ? __kmalloc+0x1d2/0x280
Apr 28 22:47:25 ceph-store5 kernel: [26693.071509] [<ffffffffc0a5b080>] ? kmem_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.078078] [<ffffffffc0a1d427>] xfs_da_grow_inode+0x37/0x90 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.085129] [<ffffffffc0a08562>] xfs_attr_shortform_to_leaf+0xa2/0x2b0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.093151] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.100199] [<ffffffffc0a5b2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.107251] [<ffffffffc0a047eb>] xfs_attr_set+0x31b/0x3e0 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.114009] [<ffffffffc0a5acc2>] xfs_xattr_set+0x42/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.120647] [<ffffffff81212a42>] generic_setxattr+0x62/0x80
Apr 28 22:47:25 ceph-store5 kernel: [26693.127107] [<ffffffff812134e1>] __vfs_setxattr_noperm+0x61/0x1a0
Apr 28 22:47:25 ceph-store5 kernel: [26693.134045] [<ffffffff812136c7>] vfs_setxattr+0xa7/0xb0
Apr 28 22:47:25 ceph-store5 kernel: [26693.145904] [<ffffffff8121376d>] ? setxattr+0x9d/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.151781] [<ffffffff812137fa>] setxattr+0x12a/0x1b0
Apr 28 22:47:25 ceph-store5 kernel: [26693.157626] [<ffffffffc0a73fcb>] ? xfs_acl_exists+0x4b/0x60 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.164599] [<ffffffffc0a74505>] ? posix_acl_access_exists+0x15/0x20 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.172451] [<ffffffffc0a5af52>] ? xfs_vn_listxattr+0xf2/0x160 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.179694] [<ffffffff811f1089>] ? __sb_start_write+0x49/0xf0
Apr 28 22:47:25 ceph-store5 kernel: [26693.186278] [<ffffffffc0a5ad30>] ? xfs_xattr_get+0x50/0x50 [xfs]
Apr 28 22:47:25 ceph-store5 kernel: [26693.193122] [<ffffffff81213a90>] SyS_fsetxattr+0xa0/0xd0
Apr 28 22:47:25 ceph-store5 kernel: [26693.199193] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
Apr 28 22:47:25 ceph-store5 kernel: [26693.206471] ---[ end trace c0a9568b8830fc8b ]---
Apr 28 22:47:25 ceph-store5 kernel: [26693.211904] XFS (bcache7): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
Apr 28 22:47:25 ceph-store5 kernel: [26693.221992] ------------[ cut here ]------------

I notice that xfs_da_grow_inode. Am I right in thinking that this was a new feature add in the 4.2 kernel?

I expect that the history of these filesystems is important. These would more have been formatted on a 3.16 kernel. Although I can't inspect these failed filesystems, I can give information about the other filesystems on the same servers, which I an 99% sure are identical:

root@ceph-store5:~# xfs_info /var/lib/ceph/osd/ceph-35
meta-data=/dev/bcache6 isize=512 agcount=4, agsize=244188661 blks
         = sectsz=512 attr=2, projid32bit=1
         = crc=1 finobt=1
data = bsize=4096 blocks=976754644, imaxpct=5
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=476930, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

 ==== Attempt to repair a filesystem (after a reboot) ====

root@ceph-store5:~# xfs_repair /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed. Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair. If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

root@ceph-store5:~# mount -o noatime,inode64 /dev/disk/by-uuid/398f78ec-c20b-468f-a2d6-1eb14804834c /var/lib/ceph/osd/ceph-39/
Killed

Dmesg output (dmesg -T):

[Fri Apr 29 10:18:45 2016] XFS (bcache2): Mounting V5 Filesystem
[Fri Apr 29 10:18:45 2016] XFS (bcache2): Starting recovery (logdev: internal)
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Not tainted 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06121aa>] xfs_buf_get_map+0x2a/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d530a>] ? xfs_free_ag_extent+0x43a/0x730 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e0 ]---
[Fri Apr 29 10:18:49 2016] XFS (bcache2): _xfs_buf_find: Block out of range: block 0x874702fa0, EOFS 0x1d1c0bea0
[Fri Apr 29 10:18:49 2016] ------------[ cut here ]------------
[Fri Apr 29 10:18:49 2016] WARNING: CPU: 6 PID: 211620 at /build/linux-lts-wily-H6pWeh/linux-lts-wily-4.2.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x27e/0x2d0 [xfs]()
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] 0000000000000000 ffff88018e11b8c8 ffffffff817bcbf8 0000000000000000
[Fri Apr 29 10:18:49 2016] ffffffffc0653300 ffff88018e11b908 ffffffff81079b5a ffff88018e11b928
[Fri Apr 29 10:18:49 2016] ffff880806ae8840 0000000000000008 0000000000000000 ffff88018e11ba18
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffff817bcbf8>] dump_stack+0x63/0x81
[Fri Apr 29 10:18:49 2016] [<ffffffff81079b5a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 10:18:49 2016] [<ffffffff81079c4a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 10:18:49 2016] [<ffffffffc061212e>] _xfs_buf_find+0x27e/0x2d0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc061235d>] xfs_buf_get_map+0x1dd/0x260 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063df26>] xfs_trans_get_buf_map+0x106/0x190 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05eb380>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d1b>] xfs_alloc_fix_freelist+0x20b/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:49 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:49 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:49 2016] ---[ end trace ce3e7a80324237e1 ]---
[Fri Apr 29 10:18:49 2016] BUG: unable to handle kernel NULL pointer dereference at 00000000000000f8
[Fri Apr 29 10:18:49 2016] IP: [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] PGD 564eec067 PUD 11408b067 PMD 0
[Fri Apr 29 10:18:49 2016] Oops: 0000 [#1] SMP
[Fri Apr 29 10:18:49 2016] Modules linked in: bridge openvswitch xfs ipmi_devintf dcdbas intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm ipmi_ssif crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev input_leds dm_multipath sb_edac scsi_dh edac_core mei_me ipmi_si lpc_ich mei ipmi_msghandler 8250_fintek wmi acpi_pad acpi_power_meter 8021q mac_hid garp mrp stp llc shpchp lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 ptp mpt2sas multipath pps_core raid_class mdio scsi_transport_sas nvme libcrc32c linear
[Fri Apr 29 10:18:49 2016] CPU: 6 PID: 211620 Comm: mount Tainted: G W 4.2.0-35-generic #40~14.04.1-Ubuntu
[Fri Apr 29 10:18:49 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 10:18:49 2016] task: ffff88080967ee00 ti: ffff88018e118000 task.ti: ffff88018e118000
[Fri Apr 29 10:18:49 2016] RIP: 0010:[<ffffffffc063e6f6>] [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] RSP: 0018:ffff88018e11ba08 EFLAGS: 00010292
[Fri Apr 29 10:18:49 2016] RAX: 0000000000000000 RBX: ffff88018e11bb18 RCX: 00000000003d2d52
[Fri Apr 29 10:18:49 2016] RDX: 00000000003d2d51 RSI: 0000000000000000 RDI: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] RBP: ffff88018e11ba28 R08: 000000000001a4f0 R09: ffff88080f8da4f0
[Fri Apr 29 10:18:49 2016] R10: ffffffffc061164a R11: ffffea001281d600 R12: ffff880afa4e4828
[Fri Apr 29 10:18:49 2016] R13: ffff880afa4e4828 R14: 0000000000000000 R15: 0000000000000008
[Fri Apr 29 10:18:49 2016] FS: 00007f42d515f880(0000) GS:ffff88080f8c0000(0000) knlGS:0000000000000000
[Fri Apr 29 10:18:49 2016] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Apr 29 10:18:49 2016] CR2: 00000000000000f8 CR3: 0000000139a06000 CR4: 00000000000406e0
[Fri Apr 29 10:18:49 2016] Stack:
[Fri Apr 29 10:18:49 2016] ffff88018e11bb18 ffff880afa4e4828 ffff88075d649ec0 ffff880164bc8800
[Fri Apr 29 10:18:49 2016] ffff88018e11bb08 ffffffffc05d5d26 ffff88015e77eb80 0000000200000000
[Fri Apr 29 10:18:49 2016] ffff88018e11ba98 ffffffff811d1075 ffff88015e77eb80 0000000000000000
[Fri Apr 29 10:18:49 2016] Call Trace:
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d5d26>] xfs_alloc_fix_freelist+0x216/0x3c0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811d1075>] ? kmem_cache_alloc+0x1e5/0x210
[Fri Apr 29 10:18:49 2016] [<ffffffffc062f2f0>] ? kmem_zone_alloc+0x70/0xf0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff813b0eed>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 10:18:49 2016] [<ffffffffc060681a>] ? xfs_perag_get+0x2a/0xc0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc05d65bd>] xfs_free_extent+0x8d/0x110 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0637ca5>] xlog_recover_process_efi+0x165/0x1a0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0639c0a>] xlog_recover_process_efis.isra.27+0x5a/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc063cc01>] xlog_recover_finish+0x21/0xb0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc06315df>] xfs_log_mount_finish+0x2f/0x50 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0628dbb>] xfs_mountfs+0x4db/0x740 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b96d>] xfs_fs_fill_super+0x36d/0x460 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f1f82>] mount_bdev+0x1a2/0x1e0
[Fri Apr 29 10:18:49 2016] [<ffffffffc062b600>] ? xfs_parseargs+0xac0/0xac0 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffffc0629e55>] xfs_fs_mount+0x15/0x20 [xfs]
[Fri Apr 29 10:18:49 2016] [<ffffffff811f2879>] mount_fs+0x39/0x160
[Fri Apr 29 10:18:49 2016] [<ffffffff8120d93b>] vfs_kern_mount+0x6b/0x120
[Fri Apr 29 10:18:49 2016] [<ffffffff8120fe8e>] do_mount+0x21e/0xcf0
[Fri Apr 29 10:18:49 2016] [<ffffffff8117db6e>] ? __get_free_pages+0xe/0x40
[Fri Apr 29 10:18:50 2016] [<ffffffff8120fb3a>] ? copy_mount_options+0x3a/0x150
[Fri Apr 29 10:18:50 2016] [<ffffffff81210c4b>] SyS_mount+0x8b/0xe0
[Fri Apr 29 10:18:50 2016] [<ffffffff817c40b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[Fri Apr 29 10:18:50 2016] Code: 13 48 85 d2 75 eb e9 5f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 56 49 89 f6 41 55 49 89 fd 41 54 53 <4c> 8b a6 f8 00 00 00 66 66 66 66 90 41 f6 44 24 78 04 74 4f 5b
[Fri Apr 29 10:18:50 2016] RIP [<ffffffffc063e6f6>] xfs_trans_binval+0x16/0x110 [xfs]
[Fri Apr 29 10:18:50 2016] RSP <ffff88018e11ba08>
[Fri Apr 29 10:18:50 2016] CR2: 00000000000000f8
[Fri Apr 29 10:18:50 2016] ---[ end trace ce3e7a80324237e2 ]---
root@ceph-store5:~#

Thanks to the magic of Ceph, I've so far managed to avoid data-loss and serious downtime, but I suspect there's a timebomb here waiting for anyone who upgrades from an older kernel to 4.2.0+.

Please let me know is there's any further information I might be able to provide to track this down.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 29 01:29 seq
 crw-rw---- 1 root audio 116, 33 Apr 29 01:29 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.19
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
MachineType: Dell Inc. PowerEdge R720xd
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-35-generic root=UUID=f022f2e6-eeba-4e27-b286-6acd2c4bd1ea ro net.ifnames=1 biosdevname=0 console=tty1 console=ttyS1,115200 initrd=ubuntu/amd64/generic/trusty/release/di-initrd BOOTIF=01-b0-83-fe-e6-17-cc nomdmonddf nomdmonisw crashkernel=384M-:128M
ProcVersionSignature: Ubuntu 4.2.0-35.40~14.04.1-generic 4.2.8-ckt5
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-35-generic N/A
 linux-backports-modules-4.2.0-35-generic N/A
 linux-firmware 1.127.22
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty uec-images
Uname: Linux 4.2.0-35-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 07/09/2014
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.4.3
dmi.board.name: 0HJK12
dmi.board.vendor: Dell Inc.
dmi.board.version: A03
dmi.chassis.asset.tag: 707
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.4.3:bd07/09/2014:svnDellInc.:pnPowerEdgeR720xd:pvr:rvnDellInc.:rn0HJK12:rvrA03:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R720xd
dmi.sys.vendor: Dell Inc.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1576599

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

apport information

[John McEleney (launchpa6)]

On one of my servers I've reverted to the 3.19.0-58-generic kernel. I still can't mount these broken filesystems.

root@ceph-store4:~# mount /var/lib/ceph/osd/ceph-2
Killed
root@ceph-store4:~# dmesg -T
...
[Fri Apr 29 11:09:27 2016] XFS (bcache4): Mounting V5 Filesystem
[Fri Apr 29 11:09:27 2016] XFS (bcache4): Starting recovery (logdev: internal)
[Fri Apr 29 11:09:29 2016] XFS (bcache4): _xfs_buf_find: Block out of range: block 0x8e8e05f48, EOFS 0x1d1c0bea0
[Fri Apr 29 11:09:29 2016] ------------[ cut here ]------------
[Fri Apr 29 11:09:29 2016] WARNING: CPU: 2 PID: 5427 at /build/linux-lts-vivid-GISjUd/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c:473 _xfs_buf_find+0x289/0x2d0 [xfs]()
[Fri Apr 29 11:09:29 2016] Modules linked in: bridge openvswitch ipmi_devintf xfs dcdbas intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd joydev sb_edac dm_multipath edac_core scsi_dh mei_me lpc_ich mei ipmi_si 8250_fintek ipmi_msghandler wmi acpi_pad mac_hid acpi_power_meter shpchp 8021q garp mrp stp llc lp parport btrfs bcache hid_generic usbhid hid raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 bnx2x raid0 mpt2sas ptp multipath raid_class pps_core mdio scsi_transport_sas nvme linear libcrc32c
[Fri Apr 29 11:09:29 2016] CPU: 2 PID: 5427 Comm: mount Not tainted 3.19.0-58-generic #64~14.04.1-Ubuntu
[Fri Apr 29 11:09:29 2016] Hardware name: Dell Inc. PowerEdge R720xd/0HJK12, BIOS 2.4.3 07/09/2014
[Fri Apr 29 11:09:29 2016] 0000000000000000 ffff88080aaeb928 ffffffff817b6c3d 0000000000000000
[Fri Apr 29 11:09:29 2016] ffffffffc0931e58 ffff88080aaeb968 ffffffff81075f3a ffff88080aaeb988
[Fri Apr 29 11:09:29 2016] ffff881008135980 0000000000000008 0000000000000000 ffff88080aaeba78
[Fri Apr 29 11:09:29 2016] Call Trace:
[Fri Apr 29 11:09:29 2016] [<ffffffff817b6c3d>] dump_stack+0x63/0x81
[Fri Apr 29 11:09:29 2016] [<ffffffff81075f3a>] warn_slowpath_common+0x8a/0xc0
[Fri Apr 29 11:09:29 2016] [<ffffffff8107602a>] warn_slowpath_null+0x1a/0x20
[Fri Apr 29 11:09:29 2016] [<ffffffffc08e9ec9>] _xfs_buf_find+0x289/0x2d0 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc08e9f3a>] xfs_buf_get_map+0x2a/0x280 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc08abbf3>] ? xfs_free_ag_extent+0x453/0x750 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc09176d1>] xfs_trans_get_buf_map+0x111/0x1a0 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc08c2690>] xfs_btree_get_bufs+0x50/0x60 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc08ac5d7>] xfs_alloc_fix_freelist+0x1a7/0x4c0 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc0907ee7>] ? kmem_zone_alloc+0x77/0x100 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffff813b01bd>] ? radix_tree_lookup+0xd/0x10
[Fri Apr 29 11:09:29 2016] [<ffffffffc08de52a>] ? xfs_perag_get+0x2a/0xd0 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc08ad01a>] xfs_free_extent+0x8a/0x110 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc0910ea0>] xlog_recover_process_efi+0x170/0x1b0 [xfs]
[Fri Apr 29 11:09:29 2016] [<ffffffffc0913024>] xlog_recover_process_efis.isra.27+0x64/0xb0 [xfs]
[Fri Apr 29 11:09:29 2016] [<fff...

[John McEleney (launchpa6)]

By wiping the log on the XFS filesystem, I can repair it, and mount it. On each drive I've repaired so far, I get the message "flfirst 118 in agf 0 too large (max = 118)". It seems likely that this filesystem corruption might be a factor.

Having said that, I don't think that a kernel Oops is ever a good response to filesystem corruption.

For the record, this is the result of one of the repairs:

root@ceph-store5:~# xfs_repair -L /dev/disk/by-uuid/7fdfdbfc-0781-43d2-b8e2-58aa913fd823
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
ALERT: The filesystem has valuable metadata changes in a log which is being
destroyed because the -L option was used.
        - scan filesystem freespace and inode maps...
Metadata corruption detected at block 0x1/0x200
flfirst 118 in agf 0 too large (max = 118)
sb_ifree 1088217, counted 1085130
sb_fdblocks 427029509, counted 423420685
        - found root inode chunk
Phase 3 - for each AG...
        - scan and clear agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 2
        - agno = 3
Phase 5 - rebuild AG headers and trees...
        - reset superblock...
Phase 6 - check inode connectivity...
        - resetting contents of realtime bitmap and summary inodes
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify and correct link counts...
done
root@ceph-store5:~# mount /var/lib/ceph/osd/ceph-33
root@ceph-store5:~# start ceph-osd id=33
ceph-osd (ceph/33) start/running, process 384759

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.6 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.6-rc6-wily/

[Christian Theune (ctheune)]

We had a similar incident, but our upgrade path was different. We're also using Ceph (Hammwer, 0.94.7)

We updated from 4.1.16 (Gentoo) to 4.4.27 (Gentoo). After updating multiple servers they came up fine. However, after around 30 minutes, multiple OSDs on multiple Hosts exposed the same problem.

> Nov 18 22:42:36 cartman07 kernel: [ 4450.796463] XFS (dm-8): _xfs_buf_find:
> Block out of range: block 0x8cbb5c7f8, EOFS 0xe8cfc000
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796469] ------------[ cut here
> ]------------
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796477] WARNING: CPU: 3 PID: 5954
> at fs/xfs/xfs_buf.c:472 _xfs_buf_find+0x2cc/0x330()
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796478] Modules linked in:
> deadline_iosched nf_log_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_log_ipv6
> nf_log_common xt_LOG xt_limit nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack
> nf_conntrack_ftp nf_conntrack x86_pkg_temp_thermal kvm_intel kvm irqbypass
> crc32c_intel nvme ixgbe acpi_cpufreq mdio dm_zero dm_thin_pool
> dm_persistent_data dm_bio_prison dm_round_robin dm_multipath xts aesni_intel
> glue_helper lrw ablk_helper cryptd aes_x86_64 fuse dm_snapshot dm_bufio
> dm_crypt dm_mirror dm_region_hash dm_log
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796517] CPU: 3 PID: 5954 Comm:
> ceph-osd Not tainted 4.4.27-gentoo #1
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796519] Hardware name:
> Thomas-Krenn.AG X9DR3-F/X9DR3-F, BIOS 3.0a 07/31/2013
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796521] 0000000000000000
> ffff88103ca1b610 ffffffff813a67b8 0000000000000000
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796524] ffffffff81c94276
> ffff88103ca1b648 ffffffff81056286 ffff88085a04ba40
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796527] 0000000000000008
> ffff88085a04ba40 0000000000000000 00000008cbb5c7f8
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796530] Call Trace:
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796538] [<ffffffff813a67b8>]
> dump_stack+0x4d/0x65
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796544] [<ffffffff81056286>]
> warn_slowpath_common+0x86/0xc0
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796548] [<ffffffff8105637a>]
> warn_slowpath_null+0x1a/0x20
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796550] [<ffffffff812ff87c>]
> _xfs_buf_find+0x2cc/0x330
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796553] [<ffffffff812ff90a>]
> xfs_buf_get_map+0x2a/0x280
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796558] [<ffffffff8132bdc6>]
> xfs_trans_get_buf_map+0x106/0x190
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796563] [<ffffffff812d8def>]
> xfs_btree_get_bufs+0x4f/0x60
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796567] [<ffffffff812c398f>]
> xfs_alloc_fix_freelist+0x20f/0x3c0
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796571] [<ffffffff812c2b14>] ?
> xfs_alloc_update_counters.isra.11+0x44/0x50
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796573] [<ffffffff813abbdd>] ?
> radix_tree_lookup+0xd/0x10
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796578] [<ffffffff812f4f5a>] ?
> xfs_perag_get+0x2a/0xb0
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796580] [<ffffffff813abbdd>] ?
> radix_tree_lookup+0xd/0x10
> Nov 18 22:42:36 cartman07 kernel: [ 4450.796583] [<ffffffff812f4f5a...

[Christian Theune (ctheune)]

For future reference, I checked all kernels up to 4.9rc6.