Btrfs crashes after checksum failure

[jeromek]

I have a btfs filesystem for /var. I get the following errors on dmesg:

[ 149.949666] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.378076] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.405156] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.405326] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.405498] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.405709] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 164.405877] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411

This inode corresponds to the file /var/lib/mythconverg/recordedseek.MYI. When I run a CHECK TABLE on this in mysql, it tells me that the table is corrupt, and when I run a REPAIR TABLE I get a kernel crash, and mysql hangs indefinitely.

[ 164.405950] ------------[ cut here ]------------
[ 164.405997] kernel BUG at /build/buildd/linux-3.2.0/mm/page-writeback.c:1896!
[ 164.406054] invalid opcode: 0000 [#1] SMP
[ 164.406071] CPU 1
[ 164.406078] Modules linked in: bnep rfcomm bluetooth parport_pc ppdev binfmt_misc xfs dm_crypt rc_dib0700_rc5 snd_hda_codec_hdmi snd_hda_codec_realtek ds3000 cx88_dvb cx88_vp3054_i2c videobuf_dvb snd_hda_intel rc_tevii_nec snd_hda_codec ext2 snd_hwdep snd_pcm tuner dvb_usb_dib0700 snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq dib7000p cx8802 dib0090 cx8800 dib7000m dib0070 snd_timer dvb_usb ir_lirc_codec cx88xx lirc_dev snd_seq_device dib8000 ir_mce_kbd_decoder dvb_core ir_sony_decoder ir_jvc_decoder tveeprom v4l2_common ir_rc6_decoder videodev ir_rc5_decoder snd dib3000mc ir_nec_decoder rc_core v4l2_compat_ioctl32 dibx000_common videobuf_dma_sg mei(C) videobuf_core btcx_risc soundcore joydev snd_page_alloc serio_raw mac_hid lm75 it87 hwmon_vid coretemp lp parport btrfs zlib_deflate libcrc32c i915 r8169 usbhid drm_kms_helper drm hid i2c_algo_bit video
[ 164.406406]
[ 164.406413] Pid: 3272, comm: mysqld Tainted: G C 3.2.0-24-generic #37-Ubuntu Gigabyte Technology Co., Ltd. H55M-D2H/H55M-S2H
[ 164.406453] RIP: 0010:[<ffffffff81121556>] [<ffffffff81121556>] clear_page_dirty_for_io+0xf6/0x100
[ 164.406487] RSP: 0018:ffff8800c377fbb8 EFLAGS: 00010246
[ 164.406506] RAX: 0100000000000822 RBX: ffffea0001d276c0 RCX: 00000000000c5f54
[ 164.406528] RDX: 00000000000c5f53 RSI: 00000000000180d0 RDI: ffffea0001d276c0
[ 164.406550] RBP: ffff8800c377fbc8 R08: ffffffffa0173437 R09: 0000000000000000
[ 164.406573] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8800cae91590
[ 164.406594] R13: 0000000000000000 R14: ffff8800c4ac6b00 R15: 0000000000000001
[ 164.406616] FS: 00007f1e09e35700(0000) GS:ffff880117c40000(0000) knlGS:0000000000000000
[ 164.406641] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 164.406659] CR2: 00007f1db01241d8 CR3: 0000000110b1a000 CR4: 00000000000006e0
[ 164.406681] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 164.406702] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 164.406725] Process mysqld (pid: 3272, threadinfo ffff8800c377e000, task ffff8800ca0dc4d0)
[ 164.406751] Stack:
[ 164.406758] ffff8800c4ac6b00 0000000000000001 ffff8800c377fc88 ffffffffa01670ec
[ 164.406784] ffff8800c377fc50 0000000000000050 ffff8800c377fc58 ffffffffa014736d
[ 164.406809] ffff880000000000 0000000000000400 0000000006294c00 0000000000000000
[ 164.406834] Call Trace:
[ 164.406859] [<ffffffffa01670ec>] prepare_pages.isra.13+0x2ac/0x360 [btrfs]
[ 164.406889] [<ffffffffa014736d>] ? btrfs_delalloc_reserve_metadata+0x1ed/0x260 [btrfs]
[ 164.406922] [<ffffffffa0167b26>] __btrfs_buffered_write+0x186/0x340 [btrfs]
[ 164.406954] [<ffffffffa0167f25>] btrfs_file_aio_write+0x245/0x350 [btrfs]
[ 164.406979] [<ffffffff810e0405>] ? call_rcu_sched+0x15/0x20
[ 164.406998] [<ffffffff81177342>] do_sync_write+0xd2/0x110
[ 164.407017] [<ffffffff812d7448>] ? apparmor_file_permission+0x18/0x20
[ 164.407039] [<ffffffff8129cc9c>] ? security_file_permission+0x2c/0xb0
[ 164.407060] [<ffffffff811778d1>] ? rw_verify_area+0x61/0xf0
[ 164.407078] [<ffffffff81177c33>] vfs_write+0xb3/0x180
[ 164.407095] [<ffffffff811780f2>] sys_pwrite64+0xa2/0xb0
[ 164.407114] [<ffffffff81664a82>] system_call_fastpath+0x16/0x1b
[ 164.407133] Code: 66 90 66 90 b8 01 00 00 00 eb 80 66 2e 0f 1f 84 00 00 00 00 00 48 89 df e8 78 fa ff ff eb 82 49 c7 c4 e0 b7 c3 81 e9 3b ff ff ff <0f> 0b 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 50 48 89 5d
[ 164.407263] RIP [<ffffffff81121556>] clear_page_dirty_for_io+0xf6/0x100
[ 164.407285] RSP <ffff8800c377fbb8>
[ 164.478985] ---[ end trace 14f7b8308b2bf295 ]---
[ 693.384926]

The btrfs filesystem has been around for a while, and I've had some problems in the past with weird errors. For example, I couldn't install a new kernel without booting into a 2.6 kernel first. I don't know whether it was the current file system that was the problem or not, as I didn't look into it too much once I had a workaround.

I'll probably end up getting rid of btrfs all round soon, so if anyone needs extra information let me know soon.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-24-generic 3.2.0-24.37
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: mythuser 2760 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfbff4000 irq 44'
   Mixer name : 'Intel IbexPeak HDMI'
   Components : 'HDA:10ec0887,1458a102,00100302 HDA:80862804,80860101,00100000'
   Controls : 52
   Simple ctrls : 22
Date: Tue May 15 21:02:30 2012
HibernationDevice: RESUME=UUID=38dfaa81-e083-4c0d-b54a-9b5bf45afb2c
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Release amd64 (20110426)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
MachineType: Gigabyte Technology Co., Ltd. H55M-D2H
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-24-generic root=/dev/mapper/system-root ro rootflags=subvol=@ enable_mtrr_cleanup mtrr_spare_reg_nr=1 quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-24-generic N/A
 linux-backports-modules-3.2.0-24-generic N/A
 linux-firmware 1.79
RfKill:

SourcePackage: linux
StagingDrivers: mei
UpgradeStatus: Upgraded to precise on 2012-05-13 (1 days ago)
dmi.bios.date: 08/20/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F3
dmi.board.name: H55M-S2H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF3:bd08/20/2010:svnGigabyteTechnologyCo.,Ltd.:pnH55M-D2H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnH55M-S2H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: H55M-D2H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Joseph Salisbury (jsalisbury)]

Do you know if this issue happened in a previous version of Ubuntu, or is this a new issue?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.4kernel[1] (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the other tags). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-rc7-precise/

[jeromek (jerome-kelleher)]

I can confirm that the bug is present in both previous kernel versions (2.6.38-11) and the most recent upstream kernel I was able to find (3.4.0-030400rc6) in the mainline builds ppa.

Failure is 100% reliable and happens the same way under all kernels.

Here is the relevant dmesg:

[ 188.712133] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.646518] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.646633] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.678699] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.678841] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.678987] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.679137] btrfs csum failed ino 339714 off 103366656 csum 2566472073 private 3685698411
[ 200.679189] ------------[ cut here ]------------
[ 200.679237] kernel BUG at /home/apw/COD/linux/mm/page-writeback.c:2117!
[ 200.679293] invalid opcode: 0000 [#1] SMP
[ 200.679311] CPU 2
[ 200.679320] Modules linked in: rfcomm bnep bluetooth parport_pc ppdev binfmt_misc xfs dm_crypt rc_dib0700_rc5 ir_lirc_codec lirc_dev ir_mce_kbd_decoder ds3000 ir_sanyo_decoder ir_sony_decoder ir_jvc_decoder ir_rc6_decoder cx88_dvb ir_rc5_decoder cx88_vp3054_i2c videobuf_dvb ir_nec_decoder snd_hda_codec_hdmi rc_tevii_nec snd_hda_codec_realtek snd_hda_intel snd_hda_codec ext2 snd_hwdep tuner snd_pcm dvb_usb_dib0700 snd_seq_midi dib0090 snd_rawmidi cx8802 dib7000p snd_seq_midi_event cx8800 snd_seq dib7000m cx88xx dib0070 dvb_usb dib8000 dvb_core tveeprom v4l2_common videodev snd_timer snd_seq_device dib3000mc snd rc_core dibx000_common videobuf_dma_sg videobuf_core microcode btcx_risc soundcore joydev snd_page_alloc serio_raw mei(C) mac_hid lm75 it87 hwmon_vid coretemp lp parport btrfs zlib_deflate libcrc32c usbhid hid r8169 i915 drm_kms_helper drm i2c_algo_bit video
[ 200.679710]
[ 200.679718] Pid: 3561, comm: mysqld Tainted: G C 3.4.0-030400rc6-generic #201205061835 Gigabyte Technology Co., Ltd. H55M-D2H/H55M-S2H
[ 200.679762] RIP: 0010:[<ffffffff8112a37e>] [<ffffffff8112a37e>] clear_page_dirty_for_io+0xee/0x100
[ 200.679798] RSP: 0018:ffff8800cabb9b88 EFLAGS: 00010246
[ 200.679816] RAX: 0100000000000822 RBX: ffffea0001c90040 RCX: 00000000000fe1d5
[ 200.679841] RDX: 00000000000fe1d4 RSI: ffff88009812ce70 RDI: ffffea0001c90040
[ 200.679864] RBP: ffff8800cabb9b98 R08: 0000000000019770 R09: ffffffffa01ac030
[ 200.679889] R10: 0000000000000001 R11: 0000000000000010 R12: ffff8800c56349e0
[ 200.679912] R13: 0000000000000000 R14: ffff880036782b00 R15: 0000000000000001
[ 200.679936] FS: 00007f37964ec700(0000) GS:ffff880117c80000(0000) knlGS:0000000000000000
[ 200.679963] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 200.679982] CR2: 00007fe85003b000 CR3: 0000000110947000 CR4: 00000000000007e0
[ 200.680005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 200.680030] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 200.680053] Process mysqld (pid: 3561, threadinfo ffff...

[jeromek (jerome-kelleher)]

Should I report this upstream?

[niedzw (niedzw)]

My Xubuntu 12.04 has black screen when i booting (Grub dont showing itself) after 2min i see this:
error: sparse file not allowed

uname -r : 3.2.0-31-generic

[ 382.136784] btrfs csum failed ino 270493 off 176128 csum 1781233088 private 2376235246
[ 1450.286775] btrfs csum failed ino 181532 off 23007232 csum 3960665211 private 1508484099
[ 1617.003196] spurious 8259A interrupt: IRQ7.
[ 2648.652609] btrfs csum failed ino 123689 off 585728 csum 2583586862 private 1666544304
[ 3067.452306] btrfs csum failed ino 17590 off 3670016 csum 1683415495 private 137833589
[ 3153.017004] btrfs csum failed ino 249253 off 954368 csum 1587001962 private 1444825505
[ 3154.160716] btrfs csum failed ino 249244 off 3846144 csum 686945458 private 207128592
[ 3155.876553] btrfs csum failed ino 171555 off 4096 csum 2611791122 private 1685378181
[ 3304.530529] btrfs csum failed ino 273558 off 14561280 csum 532961857 private 593712562
[ 3304.563429] btrfs csum failed ino 273558 off 15724544 csum 292609294 private 802495382
[ 3705.411103] btrfs csum failed ino 274801 off 20111360 csum 2071207 private 2366859259
[ 3705.445831] btrfs csum failed ino 274801 off 20627456 csum 1638738673 private 1056356611
[ 3952.303564] btrfs: sda1 checksum verify failed on 189136896 wanted 622510F6 found 7E20D838 level 0
[ 3952.303632] btrfs bad tree block start 0 189140992
[ 3952.303660] btrfs bad tree block start 0 189145088
[ 3952.303819] btrfs bad tree block start 0 189149184
[ 3952.315480] btrfs read error corrected: ino 1 off 189136896 (dev /dev/disk/by-uuid/f418e252-6b24-4d56-a7e1-ad0eaf0e5d84 sector 385792)
[ 3952.315943] btrfs read error corrected: ino 1 off 189140992 (dev /dev/disk/by-uuid/f418e252-6b24-4d56-a7e1-ad0eaf0e5d84 sector 385800)
[ 3952.316585] btrfs read error corrected: ino 1 off 189145088 (dev /dev/disk/by-uuid/f418e252-6b24-4d56-a7e1-ad0eaf0e5d84 sector 385808)
[ 3952.317050] btrfs read error corrected: ino 1 off 189149184 (dev /dev/disk/by-uuid/f418e252-6b24-4d56-a7e1-ad0eaf0e5d84 sector 385816)
[ 4090.956867] btrfs csum failed ino 33102 off 524288 csum 2737055574 private 1636010009
[ 4271.980185] i2c /dev entries driver
[ 4319.834827] btrfs csum failed ino 18725 off 77824 csum 3221564751 private 2995992982
[ 4385.557278] btrfs csum failed ino 100906 off 1581056 csum 2798862129 private 3215011344
[ 4406.498509] btrfs csum failed ino 17589 off 12062720 csum 4160799155 private 2439811368
[ 5345.963369] btrfs csum failed ino 249279 off 495616 csum 295309631 private 4002560773
[ 5361.671506] btrfs csum failed ino 18729 off 102400 csum 1481326951 private 4015704797
[ 5652.706774] btrfs csum failed ino 17589 off 14471168 csum 2218890244 private 4262042347
[ 5697.826987] btrfs csum failed ino 275701 off 6070272 csum 1257526565 private 839308723
[ 7127.204139] btrfs csum failed ino 273558 off 2715648 csum 20461928 private 3022258381
[ 7511.268798] btrfs csum failed ino 4298 off 598016 csum 2879588432 private 1100353246
[ 7513.540741] btrfs csum failed ino 18995 off 241664 csum 172638290 private 3729080707
[ 7655.985183] btrfs csum failed ino 275701 off 8290304 csum 742852851 private 212557632
[ 7656.508533] btrfs csum failed ino 275701 off 9310208 csum 1602713604 priv...