XFS: Corruption of in-memory data detected

[Chris Read]

I've got 2 NFS servers that have been running Ubuntu 12.04 for a number of years. For the last 9 months or so they've been happily running kernel 3.8.0-44 with no problem. As soon as I upgraded them to 3.13.0-53 the both experienced the same problem on an XFS filesystem when it started to have a heavy write load:

[267127.795322] XFS (sda1): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-lts-trusty-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa0250ada
[267127.811202] CPU: 10 PID: 28744 Comm: xfsaild/sda1 Tainted: G I 3.13.0-53-generic #89~precise1-Ubuntu
[267127.811206] Hardware name: IBM System x3690 X5 -[7148AC1]-/Node 1, System Card, BIOS -[MLE170CUS-1.70]- 09/23/2011
[267127.811211] 0000000000000001 ffff881016b7bb38 ffffffff8175a3e4 ffff88103f34f030
[267127.811221] ffff881023d9c000 ffff881016b7bb58 ffffffffa025317f ffffffffa0250ada
[267127.811228] ffffffff00000001 ffff881016b7bb98 ffffffffa02531ee 00000fe000200000
[267127.811236] Call Trace:
[267127.811251] [<ffffffff8175a3e4>] dump_stack+0x46/0x58
[267127.811311] [<ffffffffa025317f>] xfs_error_report+0x3f/0x50 [xfs]
[267127.811337] [<ffffffffa0250ada>] ? _xfs_buf_ioapply+0x6a/0x140 [xfs]
[267127.811363] [<ffffffffa02531ee>] xfs_corruption_error+0x5e/0x90 [xfs]
[267127.811396] [<ffffffffa02711b0>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[267127.811421] [<ffffffffa0250ada>] ? _xfs_buf_ioapply+0x6a/0x140 [xfs]
[267127.811445] [<ffffffffa0250e05>] ? xfs_bdstrat_cb+0x65/0xf0 [xfs]
[267127.811470] [<ffffffffa0250e05>] ? xfs_bdstrat_cb+0x65/0xf0 [xfs]
[267127.811494] [<ffffffffa0250ada>] _xfs_buf_ioapply+0x6a/0x140 [xfs]
[267127.811505] [<ffffffff8109fba0>] ? try_to_wake_up+0x210/0x210
[267127.811530] [<ffffffffa0250d29>] xfs_buf_iorequest+0x59/0xd0 [xfs]
[267127.811554] [<ffffffffa0250e05>] xfs_bdstrat_cb+0x65/0xf0 [xfs]
[267127.811579] [<ffffffffa0251013>] __xfs_buf_delwri_submit+0x183/0x210 [xfs]
[267127.811604] [<ffffffffa0251ac0>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[267127.811630] [<ffffffffa0251ac0>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[267127.811673] [<ffffffffa02b4f66>] xfsaild_push+0x126/0x5b0 [xfs]
[267127.811716] [<ffffffffa02b548f>] xfsaild+0x9f/0x140 [xfs]
[267127.811757] [<ffffffffa02b53f0>] ? xfsaild_push+0x5b0/0x5b0 [xfs]
[267127.811766] [<ffffffff8108fcd9>] kthread+0xc9/0xe0
[267127.811772] [<ffffffff8108fc10>] ? flush_kthread_worker+0xb0/0xb0
[267127.811781] [<ffffffff8176f928>] ret_from_fork+0x58/0x90
[267127.811787] [<ffffffff8108fc10>] ? flush_kthread_worker+0xb0/0xb0
[267127.811791] XFS (sda1): Corruption detected. Unmount and run xfs_repair
[267127.818679] XFS (sda1): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-lts-trusty-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa0250bad
[267127.818927] XFS (sda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error 5.
[267127.820914] XFS (sda1): Corruption of in-memory data detected. Shutting down filesystem
[267127.829131] XFS (sda1): Please umount the filesystem and rectify the problem(s)

I've not been able to reproduce this on demand outside our production workload. Downgrading the machines back to 3.8.0-44 has returned us to stability.

[Rasmus Borup Hansen (rbh-a)]

I' having a similar problem with trusty 3.13.0-55:

[601201.817664] ffff88016e03e000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
[601201.818224] ffff88016e03e010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
[601201.818827] ffff88016e03e020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[601201.819429] ffff88016e03e030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[601201.820013] XFS (dm-0): Internal error xfs_attr3_leaf_write_verify at line 216 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_attr_leaf.c. Caller 0xffffffffa00996f0
[601201.820910] CPU: 1 PID: 421 Comm: xfsaild/dm-0 Not tainted 3.13.0-55-generic #92-Ubuntu
[601201.820913] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.8.2 08/17/2011
[601201.820914] 0000000000000001 ffff880035cc3bd0 ffffffff81723294 ffff8802ac71d800
[601201.820918] ffff880035cc3be8 ffffffffa009d6fb ffffffffa00996f0 ffff880035cc3c20
[601201.820919] ffffffffa009d755 000000d800205500 ffff88001aa2a400 ffff880050acc2b8
[601201.820922] Call Trace:
[601201.820930] [<ffffffff81723294>] dump_stack+0x45/0x56
[601201.820964] [<ffffffffa009d6fb>] xfs_error_report+0x3b/0x40 [xfs]
[601201.820974] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.820984] [<ffffffffa009d755>] xfs_corruption_error+0x55/0x80 [xfs]
[601201.820997] [<ffffffffa00bab50>] xfs_attr3_leaf_write_verify+0x100/0x120 [xfs]
[601201.821007] [<ffffffffa00996f0>] ? _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821016] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821026] [<ffffffffa00996f0>] _xfs_buf_ioapply+0x70/0x3a0 [xfs]
[601201.821030] [<ffffffff8109abc0>] ? wake_up_state+0x20/0x20
[601201.821040] [<ffffffffa009b3d5>] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821050] [<ffffffffa009b336>] xfs_buf_iorequest+0x46/0x90 [xfs]
[601201.821060] [<ffffffffa009b3d5>] xfs_bdstrat_cb+0x55/0xb0 [xfs]
[601201.821070] [<ffffffffa009b56b>] __xfs_buf_delwri_submit+0x13b/0x210 [xfs]
[601201.821081] [<ffffffffa009c000>] ? xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821100] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821110] [<ffffffffa009c000>] xfs_buf_delwri_submit_nowait+0x20/0x30 [xfs]
[601201.821127] [<ffffffffa00facd7>] xfsaild+0x237/0x5c0 [xfs]
[601201.821145] [<ffffffffa00faaa0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[601201.821148] [<ffffffff8108b702>] kthread+0xd2/0xf0
[601201.821150] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821153] [<ffffffff81733ca8>] ret_from_fork+0x58/0x90
[601201.821155] [<ffffffff8108b630>] ? kthread_create_on_node+0x1c0/0x1c0
[601201.821158] XFS (dm-0): Corruption detected. Unmount and run xfs_repair
[601201.821581] XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 1320 of file /build/buildd/linux-3.13.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffa009971c
[601201.832780] XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem
[601201.833292] XFS (dm-0): Please umount the filesystem and rectify the problem(s)
[601201.843375] ffff880238b00000: 58 44 32 42 01 00 0e c0 00 00 00 00 00 00 00 00 XD2B............

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[juliusthms (y-thomas-7)]

This error affects on kernel 3.13.0-71, also after upgrading to kernel 3.19.0-39-generic.
After 3.13.0-71 and bugfixes on the xfs tree, it produces a similar error.
Releated to: http://oss.sgi.com/archives/xfs/2015-06/msg00382.html

[50372.723145] XFS (sdc1): Metadata corruption detected at xfs_attr3_leaf_write_verify+0xe5/0x100 [xfs], block 0x4445960c8
[50372.723337] XFS (sdc1): Unmount and run xfs_repair
[50372.723395] XFS (sdc1): First 64 bytes of corrupted metadata buffer:
[50372.723472] ffff8802fe0b2000: 00 00 00 00 00 00 00 00 fb ee 00 00 00 00 00 00 ................
[50372.723571] ffff8802fe0b2010: 10 00 00 00 00 20 0f e0 00 00 00 00 00 00 00 00 ..... ..........
[50372.723670] ffff8802fe0b2020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[50372.723769] ffff8802fe0b2030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
[50372.723870] XFS (sdc1): xfs_do_force_shutdown(0x8) called from line 1249 of file /build/linux-lts-vivid-1jarlV/linux-lts-vivid-3.19.0/fs/xfs/xfs_buf.c. Return address = 0xffffffffc02bbd22
[50372.723879] XFS (sdc1): Corruption of in-memory data detected. Shutting down filesystem
[50372.723971] XFS (sdc1): Please umount the filesystem and rectify the problem(s)
[50372.733993] XFS (sdc1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.

[juliusthms (y-thomas-7)]

Same problem with older kernels like 3.13.0-24.
The file system crashes only, there is no need to use tools like xfs_repair.

[penalvch (penalvch)]

Chris Read, thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:
apport-collect 1468039

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]