BTRFS crash with Ubuntu Server 14.04 LTS (under openstack)

[Sami Pietila]

The root filesystem became mounted as rw dmesg showing following messages:

[1759806.961342] btrfs: bdev /dev/vda1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[1759806.979607] btrfs: bdev /dev/vda1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[1759806.979771] btrfs: bdev /dev/vda1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[1759806.998053] btrfs: bdev /dev/vda1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[1759807.037591] btrfs: bdev /dev/vda1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[1759807.037704] btrfs: bdev /dev/vda1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[1759807.037800] btrfs: bdev /dev/vda1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[1759807.037894] btrfs: bdev /dev/vda1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[1759807.037988] btrfs: bdev /dev/vda1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[1759807.038081] btrfs: bdev /dev/vda1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
[1759807.038452] BTRFS error (device vdb) in btrfs_commit_transaction:1884: errno=-5 IO failure (Error while writing out transaction)
[1759807.038624] BTRFS info (device vdb): forced readonly
[1759807.038628] BTRFS warning (device vdb): Skipping commit of aborted transaction.
[1759807.038631] ------------[ cut here ]------------
[1759807.038671] WARNING: CPU: 3 PID: 177 at /build/buildd/linux-3.13.0/fs/btrfs/super.c:254 __btrfs_abort_transaction+0x50/0x110 [btrfs]()
[1759807.038673] btrfs: Transaction aborted (error -5)
[1759807.038675] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul drm glue_helper psmouse ablk_helper serio_raw cryptd joydev syscopyarea sysfillrect sysimgblt mac_hid i2c_piix4 hid_generic usbhid hid btrfs xor raid6_pq libcrc32c floppy
[1759807.038712] CPU: 3 PID: 177 Comm: btrfs-transacti Not tainted 3.13.0-32-generic #57-Ubuntu
[1759807.038715] Hardware name: Red Hat Inc. OpenStack Nova, BIOS 0.5.1 01/01/2007
[1759807.038719] 0000000000000009 ffff8803c60e1cc0 ffffffff8171bcb4 ffff8803c60e1d08
[1759807.038725] ffff8803c60e1cf8 ffffffff810676cd 00000000fffffffb ffff8803c6009000
[1759807.038729] ffff8803ca33c000 ffffffffa00e6e90 000000000000060b ffff8803c60e1d58
[1759807.038734] Call Trace:
[1759807.038750] [<ffffffff8171bcb4>] dump_stack+0x45/0x56
[1759807.038759] [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
[1759807.038764] [<ffffffff8106773c>] warn_slowpath_fmt+0x4c/0x50
[1759807.038779] [<ffffffffa004af00>] __btrfs_abort_transaction+0x50/0x110 [btrfs]
[1759807.038801] [<ffffffffa0074ebe>] cleanup_transaction+0x6e/0x290 [btrfs]
[1759807.038811] [<ffffffff810aaf00>] ? prepare_to_wait_event+0x100/0x100
[1759807.038831] [<ffffffffa0075e3a>] btrfs_commit_transaction+0x6ea/0x970 [btrfs]
[1759807.038849] [<ffffffffa0071855>] transaction_kthread+0x1b5/0x240 [btrfs]
[1759807.038866] [<ffffffffa00716a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[1759807.038873] [<ffffffff8108b3d2>] kthread+0xd2/0xf0
[1759807.038878] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.038888] [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
[1759807.038892] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.038896] ---[ end trace 122a578f827fe06a ]---
[1759807.038899] BTRFS error (device vdb) in cleanup_transaction:1547: errno=-5 IO failure
[1759807.039003] delayed_refs has NO entry
[1759807.039049] ------------[ cut here ]------------
[1759807.039071] WARNING: CPU: 3 PID: 177 at /build/buildd/linux-3.13.0/fs/btrfs/inode.c:4768 btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]()
[1759807.039073] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul drm glue_helper psmouse ablk_helper serio_raw cryptd joydev syscopyarea sysfillrect sysimgblt mac_hid i2c_piix4 hid_generic usbhid hid btrfs xor raid6_pq libcrc32c floppy
[1759807.039102] CPU: 3 PID: 177 Comm: btrfs-transacti Tainted: G W 3.13.0-32-generic #57-Ubuntu
[1759807.039104] Hardware name: Red Hat Inc. OpenStack Nova, BIOS 0.5.1 01/01/2007
[1759807.039106] 0000000000000009 ffff8803c60e1d78 ffffffff8171bcb4 0000000000000000
[1759807.039111] ffff8803c60e1db0 ffffffff810676cd ffff8803c600bcd8 ffff8803c600bcf0
[1759807.039115] ffff8803c60e1e30 ffff8803c600b800 ffff880036cc9e40 ffff8803c60e1dc0
[1759807.039120] Call Trace:
[1759807.039127] [<ffffffff8171bcb4>] dump_stack+0x45/0x56
[1759807.039132] [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
[1759807.039136] [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
[1759807.039154] [<ffffffffa0080886>] btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]
[1759807.039178] [<ffffffffa00c1cb4>] ? btrfs_first_delayed_node+0x34/0x50 [btrfs]
[1759807.039195] [<ffffffffa007157f>] btrfs_cleanup_transaction+0x42f/0x550 [btrfs]
[1759807.039211] [<ffffffffa00718a6>] transaction_kthread+0x206/0x240 [btrfs]
[1759807.039227] [<ffffffffa00716a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[1759807.039232] [<ffffffff8108b3d2>] kthread+0xd2/0xf0
[1759807.039236] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.039242] [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
[1759807.039246] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.039249] ---[ end trace 122a578f827fe06b ]---
[1759807.112804] ------------[ cut here ]------------
[1759807.112845] WARNING: CPU: 3 PID: 177 at /build/buildd/linux-3.13.0/fs/btrfs/inode.c:4768 btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]()
[1759807.112846] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul drm glue_helper psmouse ablk_helper serio_raw cryptd joydev syscopyarea sysfillrect sysimgblt mac_hid i2c_piix4 hid_generic usbhid hid btrfs xor raid6_pq libcrc32c floppy
[1759807.112866] CPU: 3 PID: 177 Comm: btrfs-transacti Tainted: G W 3.13.0-32-generic #57-Ubuntu
[1759807.112868] Hardware name: Red Hat Inc. OpenStack Nova, BIOS 0.5.1 01/01/2007
[1759807.112870] 0000000000000009 ffff8803c60e1d78 ffffffff8171bcb4 0000000000000000
[1759807.112873] ffff8803c60e1db0 ffffffff810676cd ffff8803c600bcd8 ffff8803c600bcf0
[1759807.112875] ffff8803c60e1e30 ffff8803c600b800 ffff8800bbb5d870 ffff8803c60e1dc0
[1759807.112877] Call Trace:
[1759807.112888] [<ffffffff8171bcb4>] dump_stack+0x45/0x56
[1759807.112894] [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
[1759807.112896] [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
[1759807.112906] [<ffffffffa0080886>] btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]
[1759807.112915] [<ffffffffa007157f>] btrfs_cleanup_transaction+0x42f/0x550 [btrfs]
[1759807.112924] [<ffffffffa00718a6>] transaction_kthread+0x206/0x240 [btrfs]
[1759807.112932] [<ffffffffa00716a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[1759807.112937] [<ffffffff8108b3d2>] kthread+0xd2/0xf0
[1759807.112939] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.112944] [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
[1759807.112947] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.112948] ---[ end trace 122a578f827fe06c ]---
[1759807.140484] ------------[ cut here ]------------
[1759807.140530] WARNING: CPU: 3 PID: 177 at /build/buildd/linux-3.13.0/fs/btrfs/inode.c:4768 btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]()
[1759807.140532] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul drm glue_helper psmouse ablk_helper serio_raw cryptd joydev syscopyarea sysfillrect sysimgblt mac_hid i2c_piix4 hid_generic usbhid hid btrfs xor raid6_pq libcrc32c floppy
[1759807.140550] CPU: 3 PID: 177 Comm: btrfs-transacti Tainted: G W 3.13.0-32-generic #57-Ubuntu
[1759807.140552] Hardware name: Red Hat Inc. OpenStack Nova, BIOS 0.5.1 01/01/2007
[1759807.140554] 0000000000000009 ffff8803c60e1d78 ffffffff8171bcb4 0000000000000000
[1759807.140557] ffff8803c60e1db0 ffffffff810676cd ffff8803c600bcd8 ffff8803c600bcf0
[1759807.140560] ffff8803c60e1e30 ffff8803c600b800 ffff88036ef30128 ffff8803c60e1dc0
[1759807.140562] Call Trace:
[1759807.140573] [<ffffffff8171bcb4>] dump_stack+0x45/0x56
[1759807.140579] [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
[1759807.140581] [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
[1759807.140591] [<ffffffffa0080886>] btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]
[1759807.140600] [<ffffffffa007157f>] btrfs_cleanup_transaction+0x42f/0x550 [btrfs]
[1759807.140609] [<ffffffffa00718a6>] transaction_kthread+0x206/0x240 [btrfs]
[1759807.140617] [<ffffffffa00716a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[1759807.140622] [<ffffffff8108b3d2>] kthread+0xd2/0xf0
[1759807.140624] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.140630] [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
[1759807.140632] [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[1759807.140633] ---[ end trace 122a578f827fe06d ]---
[1759841.061185] btrfs_dev_stat_print_on_error: 6 callbacks suppressed
[1759841.061202] btrfs: bdev /dev/vda1 errs: wr 17, rd 0, flush 0, corrupt 0, gen 0
[1759841.061700] btrfs: bdev /dev/vda1 errs: wr 18, rd 0, flush 0, corrupt 0, gen 0
[1759841.061869] btrfs: bdev /dev/vda1 errs: wr 19, rd 0, flush 0, corrupt 0, gen 0

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.13.0-34-generic 3.13.0-34.60
ProcVersionSignature: Ubuntu 3.13.0-34.60-generic 3.13.11.4
Uname: Linux 3.13.0-34-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 syys 13 16:20 seq
 crw-rw---- 1 root audio 116, 33 syys 13 16:20 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
CurrentDmesg:
 [ 5.227291] init: plymouth-upstart-bridge main process ended, respawning
 [ 5.242160] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
Date: Sat Sep 13 16:20:50 2014
InstallationDate: Installed on 2014-07-30 (45 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat Inc. OpenStack Nova
PciMultimedia:

ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/@/boot/vmlinuz-3.13.0-34-generic root=UUID=08c9a20d-ed96-4e91-afac-24ec261e49b8 ro rootflags=subvol=@
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-34-generic N/A
 linux-backports-modules-3.13.0-34-generic N/A
 linux-firmware 1.127.4
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Seabios
dmi.bios.version: 0.5.1
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.modalias: dmi:bvnSeabios:bvr0.5.1:bd01/01/2007:svnRedHatInc.:pnOpenStackNova:pvr2013.1.3-2.el6:cvnRedHat:ct1:cvr:
dmi.product.name: OpenStack Nova
dmi.product.version: 2013.1.3-2.el6
dmi.sys.vendor: Red Hat Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.17 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc5-utopic/

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]