btrfs remounted read-only

[Ricardo]

A server with btfrs remounted a hardware implemented RAID 5 partition read-only and wrongly assumed it was out of space on the device. After running the fsck.btrfs it was possible to mount file system again.

Error:
[2623680.594264] ------------[ cut here ]------------
[2623680.594293] WARNING: CPU: 9 PID: 26745 at /build/buildd/linux-3.13.0/fs/btrfs/super.c:254 __btrfs_abort_transaction+0x50/0x110 [btrfs]()
[2623680.594295] btrfs: Transaction aborted (error -28)
[2623680.594297] Modules linked in: intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel gpio_ich aesni_intel aes_x86_64 lrw gf128mul mac_hid glue_helper ablk_helper cryptd ipmi_si lpc_ich i7core_edac serio_raw edac_core lp parport btrfs xor raid6_pq libcrc32c igb qla2xxx i2c_algo_bit dca scsi_transport_fc ahci ptp libahci hpsa scsi_tgt pps_core
[2623680.594320] CPU: 9 PID: 26745 Comm: postgres Tainted: G I 3.13.0-45-generic #74-Ubuntu
[2623680.594322] Hardware name: HP ProLiant DL180 G6 , BIOS O20 02/01/2012
[2623680.594324] 0000000000000009 ffff8813938ab818 ffffffff81720eb6 ffff8813938ab860
[2623680.594328] ffff8813938ab850 ffffffff810677cd 00000000ffffffe4 ffff880a0383e800
[2623680.594331] ffff881403f10be0 ffffffffa01fdbe0 0000000000001697 ffff8813938ab8b0
[2623680.594335] Call Trace:
[2623680.594343] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[2623680.594349] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[2623680.594352] [<ffffffff8106783c>] warn_slowpath_fmt+0x4c/0x50
[2623680.594359] [<ffffffffa0161f00>] __btrfs_abort_transaction+0x50/0x110 [btrfs]
[2623680.594368] [<ffffffffa0173132>] __btrfs_free_extent+0x282/0xa20 [btrfs]
[2623680.594383] [<ffffffffa01d01e4>] ? btrfs_merge_delayed_refs+0x1b4/0x3d0 [btrfs]
[2623680.594392] [<ffffffffa017817a>] run_clustered_refs+0x95a/0x1060 [btrfs]
[2623680.594402] [<ffffffffa017c6f0>] btrfs_run_delayed_refs+0xe0/0x540 [btrfs]
[2623680.594414] [<ffffffffa018c79e>] btrfs_commit_transaction+0x4e/0x970 [btrfs]
[2623680.594423] [<ffffffffa01757ca>] reserve_metadata_bytes+0x23a/0x8f0 [btrfs]
[2623680.594429] [<ffffffff81012609>] ? __switch_to+0x169/0x4c0
[2623680.594438] [<ffffffffa0176aeb>] btrfs_delalloc_reserve_metadata+0x15b/0x4b0 [btrfs]
[2623680.594450] [<ffffffffa019d7d7>] __btrfs_buffered_write+0x257/0x490 [btrfs]
[2623680.594455] [<ffffffff812c3cc5>] ? SYSC_semtimedop+0x8d5/0xd50
[2623680.594467] [<ffffffffa019dc25>] btrfs_file_aio_write+0x215/0x520 [btrfs]
[2623680.594472] [<ffffffff811d8aef>] ? file_update_time+0x3f/0xd0
[2623680.594478] [<ffffffff811bd45a>] do_sync_write+0x5a/0x90
[2623680.594480] [<ffffffff811bdbe4>] vfs_write+0xb4/0x1f0
[2623680.594483] [<ffffffff811be619>] SyS_write+0x49/0xa0
[2623680.594487] [<ffffffff8173196d>] system_call_fastpath+0x1a/0x1f
[2623680.594489] ---[ end trace fe251c9e01676c97 ]---
[2623680.594492] BTRFS error (device sdb1) in __btrfs_free_extent:5783: errno=-28 No space left
[2623680.594556] BTRFS error (device sdb1) in __btrfs_free_extent:5783: errno=-28 No space left
[2623680.594557] BTRFS info (device sdb1): forced readonly
[2623680.594559] BTRFS debug (device sdb1): run_one_delayed_ref returned -28
[2623680.594560] BTRFS error (device sdb1) in btrfs_run_delayed_refs:2730: errno=-28 No space left
[2623680.605314] BTRFS debug (device sdb1): run_one_delayed_ref returned -28
[2623680.605317] BTRFS error (device sdb1) in btrfs_run_delayed_refs:2730: errno=-28 No space left
[2623680.608983] ------------[ cut here ]------------
[2623680.609001] WARNING: CPU: 11 PID: 591 at /build/buildd/linux-3.13.0/fs/btrfs/inode.c:4768 btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]()
[2623680.609002] Modules linked in: intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel gpio_ich aesni_intel aes_x86_64 lrw gf128mul mac_hid glue_helper ablk_helper cryptd ipmi_si lpc_ich i7core_edac serio_raw edac_core lp parport btrfs xor raid6_pq libcrc32c igb qla2xxx i2c_algo_bit dca scsi_transport_fc ahci ptp libahci hpsa scsi_tgt pps_core
[2623680.609020] CPU: 11 PID: 591 Comm: btrfs-transacti Tainted: G W I 3.13.0-45-generic #74-Ubuntu
[2623680.609022] Hardware name: HP ProLiant DL180 G6 , BIOS O20 02/01/2012
[2623680.609023] 0000000000000009 ffff880a063c9d10 ffffffff81720eb6 0000000000000000
[2623680.609026] ffff880a063c9d48 ffffffff810677cd ffff880a06ddc800 ffff881403027c80
[2623680.609029] ffff880a063c9dc0 ffff880a06b81000 ffff881403027cf0 ffff880a063c9d58
[2623680.609032] Call Trace:
[2623680.609036] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[2623680.609039] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[2623680.609041] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[2623680.609053] [<ffffffffa0197886>] btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]
[2623680.609065] [<ffffffffa0187c74>] btrfs_cleanup_one_transaction+0xc4/0x5a0 [btrfs]
[2623680.609076] [<ffffffffa0188234>] btrfs_cleanup_transaction+0xe4/0x550 [btrfs]
[2623680.609081] [<ffffffff8109a88a>] ? try_to_wake_up+0x1fa/0x2c0
[2623680.609092] [<ffffffffa018c86b>] ? btrfs_commit_transaction+0x11b/0x970 [btrfs]
[2623680.609097] [<ffffffff810ab0b0>] ? prepare_to_wait_event+0x100/0x100
[2623680.609108] [<ffffffffa01888a6>] transaction_kthread+0x206/0x240 [btrfs]
[2623680.609119] [<ffffffffa01886a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[2623680.609124] [<ffffffff8108b572>] kthread+0xd2/0xf0
[2623680.609126] [<ffffffff8108b4a0>] ? kthread_create_on_node+0x1c0/0x1c0
[2623680.609129] [<ffffffff817318bc>] ret_from_fork+0x7c/0xb0
[2623680.609131] [<ffffffff8108b4a0>] ? kthread_create_on_node+0x1c0/0x1c0
[2623680.609133] ---[ end trace fe251c9e01676c98 ]---
[2623680.618130] ------------[ cut here ]------------
[2623680.618154] WARNING: CPU: 11 PID: 591 at /build/buildd/linux-3.13.0/fs/btrfs/inode.c:4768 btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]()
[2623680.618156] Modules linked in: intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel gpio_ich aesni_intel aes_x86_64 lrw gf128mul mac_hid glue_helper ablk_helper cryptd ipmi_si lpc_ich i7core_edac serio_raw edac_core lp parport btrfs xor raid6_pq libcrc32c igb qla2xxx i2c_algo_bit dca scsi_transport_fc ahci ptp libahci hpsa scsi_tgt pps_core
[2623680.618177] CPU: 11 PID: 591 Comm: btrfs-transacti Tainted: G W I 3.13.0-45-generic #74-Ubuntu
[2623680.618178] Hardware name: HP ProLiant DL180 G6 , BIOS O20 02/01/2012
[2623680.618180] 0000000000000009 ffff880a063c9d10 ffffffff81720eb6 0000000000000000
[2623680.618184] ffff880a063c9d48 ffffffff810677cd ffff880a06ddc800 ffff881403027c80
[2623680.618187] ffff880a063c9dc0 ffff880a06b81000 ffff881403027cf0 ffff880a063c9d58
[2623680.618190] Call Trace:
[2623680.618196] [<ffffffff81720eb6>] dump_stack+0x45/0x56
[2623680.618200] [<ffffffff810677cd>] warn_slowpath_common+0x7d/0xa0
[2623680.618202] [<ffffffff810678aa>] warn_slowpath_null+0x1a/0x20
[2623680.618214] [<ffffffffa0197886>] btrfs_invalidate_inodes+0x1b6/0x1c0 [btrfs]
[2623680.618225] [<ffffffffa0187c74>] btrfs_cleanup_one_transaction+0xc4/0x5a0 [btrfs]
[2623680.618236] [<ffffffffa0188234>] btrfs_cleanup_transaction+0xe4/0x550 [btrfs]
[2623680.618240] [<ffffffff8109a88a>] ? try_to_wake_up+0x1fa/0x2c0
[2623680.618251] [<ffffffffa018c86b>] ? btrfs_commit_transaction+0x11b/0x970 [btrfs]
[2623680.618256] [<ffffffff810ab0b0>] ? prepare_to_wait_event+0x100/0x100
[2623680.618266] [<ffffffffa01888a6>] transaction_kthread+0x206/0x240 [btrfs]
[2623680.618277] [<ffffffffa01886a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[2623680.618281] [<ffffffff8108b572>] kthread+0xd2/0xf0
[2623680.618284] [<ffffffff8108b4a0>] ? kthread_create_on_node+0x1c0/0x1c0
[2623680.618287] [<ffffffff817318bc>] ret_from_fork+0x7c/0xb0
[2623680.618289] [<ffffffff8108b4a0>] ? kthread_create_on_node+0x1c0/0x1c0
[2623680.618291] ---[ end trace fe251c9e01676c99 ]---

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1429812

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ricardo (rjpinto)]

The partition that was changed to read only was /var, so not much hope of additional logs or crash information. Also the system is behind a proxy and apport-collect doesn't seem to know how to use it.

From what I can gather, this happen during a nightly pg_restore of one of the databases.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 272G 1.7G 269G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 40G 12K 40G 1% /dev
tmpfs 7.9G 720K 7.9G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 40G 0 40G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 272G 1.7G 269G 1% /home
/dev/sdb1 1.1T 703G 410G 64% /var

[Ricardo (rjpinto)]

/var was the umounted partition

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.0 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-rc3-vivid/

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]