btrfs fails during scrub

[Marcin Debowski]

A btrfs linear, single volume consisting of 5 drives failed to scrub:

 [ 1607.392691] BTRFS info (device sdg): scrub: started on devid 3
 [ 1607.411805] BTRFS info (device sdg): scrub: started on devid 4
 [ 1607.447140] BTRFS info (device sdg): scrub: started on devid 2
 [ 1607.473112] BTRFS info (device sdg): scrub: started on devid 5
 [ 1607.473125] BTRFS info (device sdg): scrub: started on devid 1
 [ 3200.841468] BTRFS warning (device sdg): checksum error at logical 49377465106432 on dev /dev/sdc, physical 204288065536: metadata leaf (level 0) in tree 7
 [ 3200.841474] BTRFS warning (device sdg): checksum error at logical 49377465106432 on dev /dev/sdc, physical 204288065536: metadata leaf (level 0) in tree 7
 [ 3200.841479] BTRFS error (device sdg): bdev /dev/sdc errs: wr 510, rd 510, flush 0, corrupt 2314, gen 0
 [ 3201.697768] BTRFS error (device sdg): unable to fixup (regular) error at logical 49377465106432 on dev /dev/sdc
 [ 6346.420285] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [ 6349.274210] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 17825815631468019255
 [ 6349.283534] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [ 6349.283733] BTRFS info (device sdg): scrub: not finished on devid 5 with status: -5
 [15791.355196] BTRFS warning (device sdg): checksum/header error at logical 49377465106432 on dev /dev/sdg, physical 2159525789696: metadata leaf (level 0) in tree 7
 [15791.355201] BTRFS warning (device sdg): checksum/header error at logical 49377465106432 on dev /dev/sdg, physical 2159525789696: metadata leaf (level 0) in tree 7
 [15791.355207] BTRFS error (device sdg): bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 0, gen 6
 [15791.367367] BTRFS error (device sdg): unable to fixup (regular) error at logical 49377465106432 on dev /dev/sdg
 [20535.495225] BTRFS info (device sdg): scrub: finished on devid 1 with status: 0
 [20535.805174] BTRFS info (device sdg): scrub: finished on devid 2 with status: 0
 [20539.869609] BTRFS info (device sdg): scrub: finished on devid 3 with status: 0
 [20603.754534] BTRFS info (device sdg): scrub: finished on devid 4 with status: 0
 [21834.191123] BTRFS info (device sdg): scrub: started on devid 5
 [21834.334527] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21835.759240] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 17825815631468019255
 [21835.759783] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21835.759910] BTRFS info (device sdg): scrub: not finished on devid 5 with status: -5

The following command was advised to overcome the error:
btrfs check --init-extent-tree /dev/sdg

Now, the scrub crashes as follows:

 [15791.355196] BTRFS warning (device sdg): checksum/header error at logical 49377465106432 on dev /dev/sdg, physical 2159525789696: metadata leaf (level 0) i
n tree 7
 [15791.355201] BTRFS warning (device sdg): checksum/header error at logical 49377465106432 on dev /dev/sdg, physical 2159525789696: metadata leaf (level 0) i
n tree 7
 [15791.355207] BTRFS error (device sdg): bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 0, gen 6
 [15791.367367] BTRFS error (device sdg): unable to fixup (regular) error at logical 49377465106432 on dev /dev/sdg
 [20535.495225] BTRFS info (device sdg): scrub: finished on devid 1 with status: 0
 [20535.805174] BTRFS info (device sdg): scrub: finished on devid 2 with status: 0
 [20539.869609] BTRFS info (device sdg): scrub: finished on devid 3 with status: 0
 [20603.754534] BTRFS info (device sdg): scrub: finished on devid 4 with status: 0
 [21834.191123] BTRFS info (device sdg): scrub: started on devid 5
 [21834.334527] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21835.759240] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 17825815631468019255
 [21835.759783] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21835.759910] BTRFS info (device sdg): scrub: not finished on devid 5 with status: -5
 [21917.573684] BTRFS info (device sdg): scrub: started on devid 5
 [21917.592098] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21918.932051] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 48134072074240
 [21918.935969] BTRFS error (device sdg): bad tree block start, want 49377465106432 have 17825815631468019255
 [21918.971873] BTRFS info (device sdg): scrub: not finished on devid 5 with status: -5
 [24594.433861] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
 [24633.390103] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: (null)
 [28804.075490] BTRFS info (device sdg): disk space caching is enabled
 [28804.075494] BTRFS info (device sdg): has skinny extents
 [28805.039000] BTRFS info (device sdg): bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 0, gen 6
 [28805.039009] BTRFS info (device sdg): bdev /dev/sdf errs: wr 2274065, rd 17401701, flush 0, corrupt 0, gen 1
 [28805.039016] BTRFS info (device sdg): bdev /dev/sde errs: wr 0, rd 0, flush 0, corrupt 90, gen 0
 [28805.039021] BTRFS info (device sdg): bdev /dev/sdc errs: wr 510, rd 510, flush 0, corrupt 2314, gen 0
 [28836.729425] BTRFS info (device sdg): scrub: started on devid 1
 [32648.846447] BTRFS info (device sdg): scrub: not finished on devid 1 with status: -125
 [73660.524782] BTRFS info (device sdg): disk space caching is enabled
 [73660.524785] BTRFS info (device sdg): has skinny extents
 [73661.366142] BTRFS info (device sdg): bdev /dev/sdg errs: wr 0, rd 0, flush 0, corrupt 0, gen 6
 [73661.366150] BTRFS info (device sdg): bdev /dev/sdf errs: wr 2274065, rd 17401701, flush 0, corrupt 0, gen 1
 [73661.366157] BTRFS info (device sdg): bdev /dev/sde errs: wr 0, rd 0, flush 0, corrupt 90, gen 0
 [73661.366162] BTRFS info (device sdg): bdev /dev/sdc errs: wr 510, rd 510, flush 0, corrupt 2314, gen 0
 [73661.542731] BTRFS info (device sdg): checking UUID tree
 [73690.995430] BTRFS info (device sdg): scrub: started on devid 1
 [73691.985957] BTRFS error (device sdg): tree block 49392288530432 owner 3 already locked by pid=5334, extent tree corruption detected
 [73691.985971] ------------[ cut here ]------------
 [73691.985973] BTRFS: Transaction aborted (error -117)
 [73691.986110] WARNING: CPU: 1 PID: 5334 at fs/btrfs/volumes.c:2987 btrfs_remove_chunk+0x6bc/0x700 [btrfs]
 [73691.986112] Modules linked in: nls_iso8859_1 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg intel_rapl_msr snd_hda_codec snd_hda_core snd_hwdep mei_hdcp snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel snd_seq_midi snd_seq_midi_event kvm snd_rawmidi crct10dif_pclmul ghash_clmulni_intel cryptd joydev rapl intel_cstate snd_seq input_leds efi_pstore i915 snd_seq_device snd_timer mxm_wmi at24 snd drm_kms_helper cec soundcore rc_core mei_me i2c_algo_bit fb_sys_fops syscopyarea mei sysfillrect sysimgblt mac_hid intel_smartconnect sch_fq_codel nct6775 hwmon_vid coretemp parport_pc ppdev lp parport drm ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid crc32_pclmul mpt3sas ahci xhci_pci libahci lpc_ich xhci_pci_renesas raid_class r8169 i2c_i801 i2c_smbus realtek scsi_transport_sas wmi video
 [73691.986190] CPU: 1 PID: 5334 Comm: btrfs-cleaner Not tainted 5.8.0-59-generic #66~20.04.1-Ubuntu
 [73691.986192] Hardware name: MSI MS-7835/C847MS-E33 (MS-7835), BIOS V1.1 03/07/2013
 [73691.986256] RIP: 0010:btrfs_remove_chunk+0x6bc/0x700 [btrfs]
 [73691.986262] Code: 8b 57 50 f0 48 0f ba aa 40 0a 00 00 02 8b 45 a0 72 1d 83 f8 fb 74 43 83 f8 e2 74 3e 89 c6 48 c7 c7 28 19 57 c0 e8 dc 41 03 ce <0f> 0b 8b 45 a0 89 c1 ba ab 0b 00 00 4c 89 ff 89 45 a0 48 c7 c6 b0
 [73691.986265] RSP: 0018:ffffc1ed4091fd68 EFLAGS: 00010286
 [73691.986269] RAX: 0000000000000000 RBX: 0000000040000000 RCX: 0000000000000027
 [73691.986271] RDX: 0000000000000027 RSI: 0000000000000086 RDI: ffff9e2617318cd8
 [73691.986274] RBP: ffffc1ed4091fe18 R08: ffff9e2617318cd0 R09: 0000000000000004
 [73691.986276] R10: 0000000000000000 R11: 0000000000000001 R12: ffff9e25fa7c8800
 [73691.986278] R13: ffff9e256904bc60 R14: ffff9e26099d8388 R15: ffff9e257895bb60
 [73691.986282] FS: 0000000000000000(0000) GS:ffff9e2617300000(0000) knlGS:0000000000000000
 [73691.986285] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 [73691.986288] CR2: 0000560b1491bd90 CR3: 0000000135c0a006 CR4: 00000000000606e0
 [73691.986291] Call Trace:
 [73691.986363] btrfs_delete_unused_bgs+0x6f5/0x790 [btrfs]
 [73691.986416] cleaner_kthread+0x124/0x130 [btrfs]
 [73691.986424] kthread+0x114/0x150
 [73691.986473] ? refcount_add_not_zero.constprop.0+0x50/0x50 [btrfs]
 [73691.986477] ? kthread_park+0x90/0x90
 [73691.986484] ret_from_fork+0x22/0x30
 [73691.986490] ---[ end trace 6ea90edff35b62cd ]---
 [73691.986496] BTRFS: error (device sdg) in btrfs_remove_chunk:2987: errno=-117 Filesystem corrupted
[73691.986502] BTRFS info (device sdg): forced readonly
 [73691.986540] ------------[ cut here ]------------
 [73691.986605] WARNING: CPU: 1 PID: 5334 at fs/btrfs/space-info.h:116 btrfs_block_rsv_release+0x2f4/0x300 [btrfs]
 [73691.986606] Modules linked in: nls_iso8859_1 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg intel_rapl_msr snd_hda_codec snd_hda_core snd_hwdep mei_hdcp snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp kvm_intel snd_seq_midi snd_seq_midi_event kvm snd_rawmidi crct10dif_pclmul ghash_clmulni_intel cryptd joydev rapl intel_cstate snd_seq input_leds efi_pstore i915 snd_seq_device snd_timer mxm_wmi at24 snd drm_kms_helper cec soundcore rc_core mei_me i2c_algo_bit fb_sys_fops syscopyarea mei sysfillrect sysimgblt mac_hid intel_smartconnect sch_fq_codel nct6775 hwmon_vid coretemp parport_pc ppdev lp parport drm ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid crc32_pclmul mpt3sas ahci xhci_pci libahci lpc_ich xhci_pci_renesas raid_class r8169 i2c_i801 i2c_smbus realtek scsi_transport_sas wmi video
 [73691.986669] CPU: 1 PID: 5334 Comm: btrfs-cleaner Tainted: G W 5.8.0-59-generic #66~20.04.1-Ubuntu
 [73691.986671] Hardware name: MSI MS-7835/C847MS-E33 (MS-7835), BIOS V1.1 03/07/2013
 [73691.986733] RIP: 0010:btrfs_block_rsv_release+0x2f4/0x300 [btrfs]
 [73691.986737] Code: b2 6a 2b ce 49 8b 06 48 85 c0 75 d8 4c 8b 6d c0 4c 8b 75 b8 4c 8b 65 b0 4c 8b 5d a8 e9 af fe ff ff 4c 39 f0 0f 83 b6 fe ff ff <0f> 0b 31 c0 e9 b0 fe ff ff 0f 1f 00 66 66 66 66 90 48 85 f6 74 23
 [73691.986740] RSP: 0018:ffffc1ed4091fcf0 EFLAGS: 00010287
 [73691.986743] RAX: 000000000005c000 RBX: ffff9e26099d8000 RCX: 0000000000000001
 [73691.986746] RDX: 0000000000000001 RSI: ffff9e26099d8170 RDI: ffff9e25f51cfc00
 [73691.986748] RBP: ffffc1ed4091fd48 R08: ffff9e2617318cd0 R09: 0000000000000004
 [73691.986750] R10: 0000000000000000 R11: 0000000000060000 R12: ffff9e25f51cfc00
 [73691.986752] R13: 0000000000000000 R14: 0000000000060000 R15: 0000000000000000
 [73691.986756] FS: 0000000000000000(0000) GS:ffff9e2617300000(0000) knlGS:0000000000000000
 [73691.986759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 [73691.986761] CR2: 0000560b1491bd90 CR3: 0000000135c0a006 CR4: 00000000000606e0
 [73691.986763] Call Trace:
 [73691.986818] btrfs_trans_release_chunk_metadata+0x35/0x50 [btrfs]
 [73691.986876] btrfs_create_pending_block_groups+0x140/0x240 [btrfs]
 [73691.986928] __btrfs_end_transaction+0x3f/0x1b0 [btrfs]
 [73691.986978] btrfs_end_transaction+0x10/0x20 [btrfs]
 [73691.987035] btrfs_delete_unused_bgs+0x339/0x790 [btrfs]
 [73691.987085] cleaner_kthread+0x124/0x130 [btrfs]
 [73691.987090] kthread+0x114/0x150
 [73691.987138] ? refcount_add_not_zero.constprop.0+0x50/0x50 [btrfs]
 [73691.987142] ? kthread_park+0x90/0x90
 [73691.987148] ret_from_fork+0x22/0x30
 [73691.987153] ---[ end trace 6ea90edff35b62ce ]---
 [73691.993753] BTRFS warning (device sdg): failed setting block group ro: -30
 [73691.993759] BTRFS info (device sdg): scrub: not finished on devid 1 with status: -30

# apt list|grep btrfs

apt-btrfs-snapshot/focal,focal 3.5.3 all
btrfs-compsize/focal 1.3-2 amd64
btrfs-heatmap/focal,focal,now 8-1 all [installed]
btrfs-progs/focal,now 5.4.1-2 amd64 [installed]
btrfs-progs/focal 5.4.1-2 i386
btrfsmaintenance/focal,focal 0.4.2-1 all
golang-github-containerd-btrfs-dev/focal,focal 0.0~git20200117.1539353-1 all
libblockdev-btrfs-dev/focal 2.23-2ubuntu3 amd64
libblockdev-btrfs-dev/focal 2.23-2ubuntu3 i386
libblockdev-btrfs2/focal 2.23-2ubuntu3 amd64
libblockdev-btrfs2/focal 2.23-2ubuntu3 i386
libbtrfs-dev/focal 5.4.1-2 amd64
libbtrfs-dev/focal 5.4.1-2 i386
libbtrfs0/focal 5.4.1-2 amd64
libbtrfs0/focal 5.4.1-2 i386
libbtrfsutil-dev/focal 5.4.1-2 amd64
libbtrfsutil-dev/focal 5.4.1-2 i386
libbtrfsutil1/focal 5.4.1-2 amd64
libbtrfsutil1/focal 5.4.1-2 i386
monitoring-plugins-btrfs/focal,focal 11-2 all
munin-plugins-btrfs/focal,focal 11-2 all
python3-btrfs/focal,focal,now 11-2 all [installed,automatic]
python3-btrfsutil/focal 5.4.1-2 amd64
python3-btrfsutil/focal 5.4.1-2 i386
udisks2-btrfs/focal 2.8.4-1ubuntu1 amd64
udisks2-btrfs/focal 2.8.4-1ubuntu1 i386

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1937380/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]