Unable to mount btrfs RAID 1 filesystem after reboot - Error - device total_bytes should be at most X but found Y

Bug #1931790 reported by Ravindran K
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
btrfs (Ubuntu)
Confirmed
Undecided
Unassigned
btrfs-progs (Ubuntu)
Confirmed
Undecided
Unassigned
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Hi,

I'm unable to mount my newly created btrfs RAID 1 filesystem after reboot..I performed the below steps with my 2 X 1 TB drives:

* created a new btrfs filesystem
mkfs.btrfs /dev/sdd1

* mounted it to /mnt/datanew
mount -t btrfs /dev/sdd1 /mnt/datanew

* copied over data from my previous NTFS volume on /dev/sde1 (which I intended to add later as a mirror) - took 3.5 hrs

* added a new volume
mkfs.btrfs device add /dev/sde1

* started balance and convert to raid 1 - took 8 to 10 hours..overnight

btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/datanew

* rebooted next morning

I seeing below errors:
[ 792.045980] BTRFS info (device sdd1): allowing degraded mounts
[ 792.045988] BTRFS info (device sdd1): disk space caching is enabled
[ 792.045990] BTRFS info (device sdd1): has skinny extents
[ 792.047498] BTRFS error (device sdd1): device total_bytes should be at most 1000201841152 but found 1000203804672
[ 792.049207] BTRFS error (device sdd1): failed to read chunk tree: -22
[ 792.051525] BTRFS error (device sdd1): open_ctree failed
* Manual mount has below error additonally
mount: /mnt/datanew: wrong fs type, bad option, bad superblock on /dev/sde1, missing codepage or helper program, or other error.

* Verified the data seems intact (apparently no errors), superblock seems fine.. tried btrfs rescue commands.. no issues found to fix

* Found several hits on search but not many solutions..except "btrfs filesystem resize max /" which doesnt work as I'm unable to mount in the first place..
btrfs filesystem resize max /mnt/datanew/
ERROR: not a btrfs filesystem: /mnt/datanew/

Any suggestions? Thanks

ProblemType: Bug
DistroRelease: Ubuntu 21.04
Package: ubuntu-release-upgrader-core 1:21.04.13
ProcVersionSignature: Ubuntu 5.11.0-17.18-generic 5.11.12
Uname: Linux 5.11.0-17-generic x86_64
ApportVersion: 2.20.11-0ubuntu65.1
Architecture: amd64
CasperMD5CheckResult: pass
CrashDB: ubuntu
Date: Sun Jun 13 11:17:02 2021
InstallationDate: Installed on 2021-05-16 (27 days ago)
InstallationMedia: Kubuntu 21.04 "Hirsute Hippo" - Release amd64 (20210420)
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_IN:en
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_IN
 SHELL=/bin/bash
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Ravindran K (ravindran-k) wrote :
description: updated
Revision history for this message
Ravindran K (ravindran-k) wrote (last edit ):
Download full text (4.5 KiB)

I think the key is :
[ 792.047498] BTRFS error (device sdd1): device total_bytes should be at most 1000201841152 but found 1000203804672
[ 792.049207] BTRFS error (device sdd1): failed to read chunk tree: -22
[ 792.051525] BTRFS error (device sdd1): open_ctree failed

Additional info:
# lsblk -e7
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 19.5G 0 part
├─sda2 8:2 0 620M 0 part
├─sda3 8:3 0 48.8G 0 part /
└─sda4 8:4 0 8G 0 part [SWAP]
sdb 8:16 0 232.9G 0 disk
├─sdb1 8:17 0 200M 0 part
└─sdb2 8:18 0 37.3G 0 part
sdc 8:32 0 465.8G 0 disk
├─sdc1 8:33 0 200M 0 part
├─sdc2 8:34 0 97.7G 0 part
├─sdc3 8:35 0 326.9G 0 part
└─sdc4 8:36 0 41.1G 0 part
sdd 8:48 0 931.5G 0 disk
└─sdd1 8:49 0 931.5G 0 part
sde 8:64 0 931.5G 0 disk
└─sde1 8:65 0 931.5G 0 part

# blockdev --getsize64 /dev/sdd1
1000201841152
# blockdev --getsize64 /dev/sde1
1000201841152

#btrfs fi show
Label: 'DataNew' uuid: ae9d1623-f910-4dec-a88f-f762dbdba35e
        Total devices 2 FS bytes used 889.78GiB
        devid 1 size 931.51GiB used 892.06GiB path /dev/sdd1
        devid 2 size 931.51GiB used 892.06GiB path /dev/sde1

# btrfs check /dev/sdd1
Opening filesystem to check...
Checking filesystem on /dev/sdd1
UUID: ae9d1623-f910-4dec-a88f-f762dbdba35e
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 955392856064 bytes used, no error found
total csum bytes: 931797364
total tree bytes: 1232355328
total fs tree bytes: 226492416
total extent tree bytes: 30195712
btree space waste bytes: 79534096
file data blocks allocated: 954160500736
 referenced 954160500736

# btrfs check /dev/sde1
Opening filesystem to check...
Checking filesystem on /dev/sde1
UUID: ae9d1623-f910-4dec-a88f-f762dbdba35e
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 955392856064 bytes used, no error found
total csum bytes: 931797364
total tree bytes: 1232355328
total fs tree bytes: 226492416
total extent tree bytes: 30195712
btree space waste bytes: 79534096
file data blocks allocated: 954160500736
 referenced 954160500736

# btrfs inspect-internal dump-super /dev/disk/by-uuid/ae9d1623-f910-4dec-a88f-f762dbdba35e
superblock: bytenr=65536, device=/dev/disk/by-uuid/ae9d1623-f910-4dec-a88f-f762dbdba35e
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xbbd62202 [match]
bytenr 65536
flags 0x1
                        ( WRITTEN )
magic _BHRfS_M [match]
fsid ae9d1623-f910-4dec-a88f-f762dbdba35e
metadata_uuid ae9d1623-f910-4dec-a88f-f762dbdba35e
label Da...

Read more...

description: updated
summary: - Unable to mount btrfs RAID 1 filesystem after reboot
+ Unable to mount btrfs RAID 1 filesystem after reboot - Error - device
+ total_bytes should be at most X but found Y
Revision history for this message
Ravindran K (ravindran-k) wrote :

I was about to give up ..but I was trying older/newer kernels/ other distros..

Tried multiple but one last try on Linux Mint opened the doors..

I was surprised to see the filesystem mounted fine with no errors..

BTRFS info (device sdd1): disk space caching is enabled
[ 396.259897] BTRFS info (device sdd1): has skinny extents

mint@mint:~$ uname -a
Linux mint 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

mint@mint:~$ btrfs version
btrfs-progs v5.4.1

mint@mint:~$ sudo btrfs inspect-internal dump-super /dev/disk/by-uuid/ae9d1623-f910-4dec-a88f-f762dbdba35e
superblock: bytenr=65536, device=/dev/disk/by-uuid/ae9d1623-f910-4dec-a88f-f762dbdba35e
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x2564561f [match]
bytenr 65536
flags 0x1
   ( WRITTEN )
magic _BHRfS_M [match]
fsid ae9d1623-f910-4dec-a88f-f762dbdba35e
metadata_uuid ae9d1623-f910-4dec-a88f-f762dbdba35e
label DataNew
generation 10270
root 944046407680
sys_array_size 129
chunk_root_generation 10270
root_level 0
chunk_root 944923213824
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 2000405643264
bytes_used 955398033408
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 2
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x161
   ( MIXED_BACKREF |
     BIG_METADATA |
     EXTENDED_IREF |
     SKINNY_METADATA )
cache_generation 10270
uuid_tree_generation 10270
dev_item.uuid 67a4b867-0d9d-498b-a311-19f7b9bc5c77
dev_item.fsid ae9d1623-f910-4dec-a88f-f762dbdba35e [match]
dev_item.type 0
dev_item.total_bytes 1000201838592
dev_item.bytes_used 957811261440
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0

Totally surprised that dev_item.total_bytes is different here...

I will recheck the btrfs version on my Ubuntu hirsute..

Any thoughts..? For now, glad I haven't lost any data.. (yet)

Revision history for this message
Ravindran K (ravindran-k) wrote :

As a workaround, I installed the 5.4 kernel using mainline and my raid1 btrfs volume seems to be mounting fine..The issue is present even with latest mainline 5.13 kernel..so definitely more people should be seeing this.. as and when probably they create/convert to raid 1..

@:~$ uname -a
Linux monster 5.4.126-0504126-generic #202106160800 SMP Wed Jun 16 12:12:41 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
@:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.04
Release: 21.04
Codename: hirsute
@:~$ mount | grep btrfs
/dev/sdd1 on /mnt/datanew type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

Revision history for this message
Ravindran K (ravindran-k) wrote :

I tried to update Filesystem type (from 42 SFS to 83), which messed one of the disks (luckily) and had to rebuild the RAID.. post which I found that this specific issue is now gone. I can mount fine on any kernel version..

Please feel free to close this.. I hope this help someone who is facing this issue or faces this in future..

Thank you.

Revision history for this message
Łukasz (puksi) wrote :

I have the very same problem, raid1 partition does not mount after upgrade from 5.8 to 5.11 kernel.

[ 202.604064] BTRFS error (device sdd): device total_bytes should be at most 6000606183424 but found 6001175126016

Revision history for this message
Ravindran K (ravindran-k) wrote (last edit ):

Hi Lukasz,

Unfortunately I could not find much alternatives/solutions in multiple forums..

The best and easy workaround (if you are prepared to wait infinely: as noone responds to these one off issues) would be to install the mainline tool and install a older version of kernel:
https://ubuntuhandbook.org/index.php/2020/08/mainline-install-latest-kernel-ubuntu-linux-mint/

This will most likely work as it has worked for multiple people..Make sure you choose old version of kernel like 5.4 (that worked for me). Useful Tip:Try booting using Ubuntu Mint USB to see if you btrfs volume readily mounts.

The other alternatuve solution which I discovered the hardway is to break one of the disks (In my case, I rebuilt both disks one after the other due to some issues) and re-add it and rebuild to see if that solves..

The other best solution would be to delete both the raid devices and create a new raid 1 volume from scratch while booted on the new kernel. ofcourse this needs enough space and a backup which you can restore.

I reiterate, please backup if you data is valuable before you try anything as inadvertently you might lose data while troubleshooting this issue... All the best ..keep us posted :)

Regards
Ravindran

Revision history for this message
Łukasz (puksi) wrote (last edit ):

I tried as in[1]:

  btrfs filesystem resize max /path/to/mount

It did resize, but it is still not working with 5.11 kernel, had to go back to 5.8.

> The other alternatuve solution which I discovered the hardway is to break one of the disks (In my case, I rebuilt both disks one after he other due to some issues) and re-add it and rebuild to see if that solves..

Can you write specific steps? I understand that needs to happened on working kernel. Surprisingly I have two raid1 devices. The root is working fine, however the other one mounted at /mnt is not.

https://bugs.archlinux.org/task/69778

Revision history for this message
Ravindran K (ravindran-k) wrote :

I messed with the partition tables on one of the disks and then followed some steps to remove the failed one and recreated the new partition and added..all in older kernel..

Same thing happened for the other disk and after rebuilding again... effectively redoing the raid. Then, surprisingly ,booting the new kernel worked fine. Magically 😃

Sorry I don't have the steps handy :(

Revision history for this message
Łukasz (puksi) wrote :

I managed to fix my problem recreating file system on disks.

I wrote blog post about it:
https://zarnowiecki.pl/posts/btrfs-raid1-does-not-mount/

Revision history for this message
Ravindran K (ravindran-k) wrote :
Download full text (3.4 KiB)

Great article. Will be useful for more people who are going to run into
this issue.

On Wed, Aug 18, 2021 at 4:15 PM Łukasz <email address hidden> wrote:

> I managed to fix my problem recreating file system on disks.
>
> I wrote blog post about it:
> https://zarnowiecki.pl/posts/btrfs-raid1-does-not-mount/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1931790
>
> Title:
> Unable to mount btrfs RAID 1 filesystem after reboot - Error - device
> total_bytes should be at most X but found Y
>
> Status in Ubuntu:
> New
> Status in btrfs package in Ubuntu:
> New
> Status in btrfs-progs package in Ubuntu:
> New
>
> Bug description:
> Hi,
>
> I'm unable to mount my newly created btrfs RAID 1 filesystem after
> reboot..I performed the below steps with my 2 X 1 TB drives:
>
> * created a new btrfs filesystem
> mkfs.btrfs /dev/sdd1
>
> * mounted it to /mnt/datanew
> mount -t btrfs /dev/sdd1 /mnt/datanew
>
> * copied over data from my previous NTFS volume on /dev/sde1 (which I
> intended to add later as a mirror) - took 3.5 hrs
>
> * added a new volume
> mkfs.btrfs device add /dev/sde1
>
> * started balance and convert to raid 1 - took 8 to 10
> hours..overnight
>
> btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/datanew
>
> * rebooted next morning
>
> I seeing below errors:
> [ 792.045980] BTRFS info (device sdd1): allowing degraded mounts
> [ 792.045988] BTRFS info (device sdd1): disk space caching is enabled
> [ 792.045990] BTRFS info (device sdd1): has skinny extents
> [ 792.047498] BTRFS error (device sdd1): device total_bytes should be
> at most 1000201841152 but found 1000203804672
> [ 792.049207] BTRFS error (device sdd1): failed to read chunk tree: -22
> [ 792.051525] BTRFS error (device sdd1): open_ctree failed
> * Manual mount has below error additonally
> mount: /mnt/datanew: wrong fs type, bad option, bad superblock on
> /dev/sde1, missing codepage or helper program, or other error.
>
> * Verified the data seems intact (apparently no errors), superblock
> seems fine.. tried btrfs rescue commands.. no issues found to fix
>
> * Found several hits on search but not many solutions..except "btrfs
> filesystem resize max /" which doesnt work as I'm unable to mount in the
> first place..
> btrfs filesystem resize max /mnt/datanew/
> ERROR: not a btrfs filesystem: /mnt/datanew/
>
> Any suggestions? Thanks
>
> ProblemType: Bug
> DistroRelease: Ubuntu 21.04
> Package: ubuntu-release-upgrader-core 1:21.04.13
> ProcVersionSignature: Ubuntu 5.11.0-17.18-generic 5.11.12
> Uname: Linux 5.11.0-17-generic x86_64
> ApportVersion: 2.20.11-0ubuntu65.1
> Architecture: amd64
> CasperMD5CheckResult: pass
> CrashDB: ubuntu
> Date: Sun Jun 13 11:17:02 2021
> InstallationDate: Installed on 2021-05-16 (27 days ago)
> InstallationMedia: Kubuntu 21.04 "Hirsute Hippo" - Release amd64
> (20210420)
> PackageArchitecture: all
> ProcEnviron:
> LANGUAGE=en_IN:en
> TERM=xterm-256color
> PATH=(custom, no user)
> LANG=en_IN
> SHELL=/bin/bash
> SourcePackage: ubuntu-r...

Read more...

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in btrfs (Ubuntu):
status: New → Confirmed
Changed in btrfs-progs (Ubuntu):
status: New → Confirmed
Changed in ubuntu:
status: New → Confirmed
Revision history for this message
Andreas Mewes (glio73) wrote :
Download full text (4.8 KiB)

Hi, I have the same problem, raid1 partition does not mount after upgrade from 5.8 to 5.11 kernel.
(Ubuntu 20.04LTS > 21.04)
Mount is ok with Kernel up to 5.10. (SystemRescue-USB)

BTRFS error (device sdb): device total_bytes should be at most 4000780910592 but found 4000785960960
Why the difference of 5050368 bytes (9864 sectors) ?
Łukasz had a difference of 568942592 bytes (1111216 sectors)

sdb is the only Harddrive with both. 512 bytes Sector size (logical/physical)!

sda: root/boot SDD
sdb,sdc,sdd,sde: RAID1 btrfspool
sdf: clean 16TB btrfs HDD

Disk /dev/sda: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 850
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sda1 2048 10239 8192 4M BIOS boot
/dev/sda2 10240 1034239 1024000 500M EFI System
/dev/sda3 1034240 488397134 487362895 232.4G Linux filesystem

Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: HGST HUS726T4TAL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sdb1 10240 7814035455 7814025216 3.6T Linux filesystem
/dev/sdb2 2048 10239 8192 4M BIOS boot

Disk /dev/sdc: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: WDC WD8003FRYZ-0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sdc1 10240 15628053134 15628042895 7.3T Linux filesystem
/dev/sdc2 2048 10239 8192 4M BIOS boot

Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD40EFRX-68N
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sdd1 10240 7814035455 7814025216 3.6T Linux filesystem
/dev/sdd2 2048 10239 8192 4M BIOS boot

Disk /dev/sde: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: WDC WD4003FFBX-6
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sde1 10240 7814035455 7814025216 3.6T Linux filesystem
/dev/sde2 2048 10239 8192 4M BIOS boot

Disk /dev/sdf: 14.55 TiB, 16000900661248 bytes, 31251759104 sectors
Disk model: TOSHIBA MG08ACA1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sdf1 2048 31251759070 31251757023 14.6T Linux filesystem

Comparision (extract) between sdb1 and sdd1, sde1:
[root@sysrescue ~...

Read more...

Revision history for this message
Jacob Anawalt (jlanawalt) wrote :

I also experienced this upgrading from Ubuntu 20.04LTS to 22.04. Fortunately the btrfs was used for network file storage and backups commenting out those mount points I could still boot. Unfortunately the old 5.4.0-126-generic kernel would not boot my LVM root. Neither would 5.4.214 or 5.5.19. Fortunately 5.10.120-0510120-generic did and could still mount my btrfs.

For a precaution I did a scrub and a balance, neither helped, and then I ran btrfs filesystem resize 4:max /btrfs-admin/storage and then rebooted a final time with success on 5.15-0-48-generic.

I had to give the id of 4 because I have migrated from older disks over time and no longer had a device id of 1. Also the ubuntu-mainline-kernel.sh program was a great help in pulling down and installing older kernels to get me to the 5.10 that worked.

Anyone upgrading from 20.04 to 22.04 with an LVM root and some BTRFS storage, prepare or beware.

Benjamin Drung (bdrung)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

Does this issue also happen on newer upstream kernels?

Revision history for this message
Esko Luontola (luontola) wrote (last edit ):
Download full text (6.7 KiB)

TLDR:
I had the "device total_bytes should be at most X but found Y" error which prevented me from mounting the btrfs filesystem. It was apparently caused by a HDD's usable disk space/partition size shrinking by about 1MB (perhaps the HDD detected bad sectors). "btrfs check" found no errors. I had to downgrade to kernel 5.10 (with ubuntu-mainline-kernel.sh) to be able to mount the filesystem. After it was mounted, I was able to fix the filesystem by running "btrfs filesystem resize max". Then I could again boot to kernel 5.15.

What happened:

I had been running btrfs in RAID1 for many years. Originally /dev/sdb1 and /dev/sdc1 were part of the btrfs volume. After getting low on disk space, I repartitioned /dev/sda1 and /dev/sdd1, added them to the same btrfs volume, and balanced the filesystem. All 4 disks were the same size and model. (I had bought all 4 disks at the same time, but originally used sda and sdd just for some backups.)

But the next time I rebooted the system, the partition sda1 had disappeared and the btrfs volume failed to mount due to a missing device. I recreated the sda1 partition using the same parted command as before ("mkpart my-btrfs btrfs 4MiB 100%"). Now btrfs no longer warned about a missing device. I ran "btrfs check" and it found no errors in the filesystem:

# btrfs check /dev/sdb1
Opening filesystem to check...
Checking filesystem on /dev/sdb1
UUID: 1ed4397c-bac2-43b7-8e00-fec7bfcecee6
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 8583317286912 bytes used, no error found
total csum bytes: 8369873676
total tree bytes: 10528210944
total fs tree bytes: 1233027072
total extent tree bytes: 295829504
btree space waste bytes: 790336946
file data blocks allocated: 10413926699008
 referenced 10416270606336

However, trying to mount the volume would fail:

# mount /mnt/data
mount: /mnt/data: wrong fs type, bad option, bad superblock on /dev/sdd1, missing codepage or helper program, or other error.

# tail /var/log/kern.log
Feb 1 16:38:24 omega kernel: [11659.827098] BTRFS info (device sdb1): using crc32c (crc32c-generic) checksum algorithm
Feb 1 16:38:24 omega kernel: [11659.827120] BTRFS info (device sdb1): disk space caching is enabled
Feb 1 16:38:24 omega kernel: [11659.827123] BTRFS info (device sdb1): has skinny extents
Feb 1 16:38:24 omega kernel: [11659.920248] BTRFS error (device sdb1): device total_bytes should be at most 8001557626880 but found 8001558675456
Feb 1 16:38:24 omega kernel: [11659.920318] BTRFS error (device sdb1): failed to read chunk tree: -22
Feb 1 16:38:24 omega kernel: [11659.934266] BTRFS error (device sdb1): open_ctree failed

I was using kernel 5.15.0. I downgraded to kernel 5.10.130 using ubuntu-mainline-kernel.sh, and with that kernel I was able to mount the volume.

Looking at the disk sizes, the disks sdb, sdc and sdd were 8001563222016B each. But sda was about 1MB smaller than the others at 8001562140160B. sda's SMART data didn't report any reallocated sectors or similar,...

Read more...

Revision history for this message
Esko Luontola (luontola) wrote :

btrfs check in btrfs-progs-5.19.1 should be able to detect and fix this. (I'm running Ubuntu 22.04.3 LTS and it has an old version, btrfs-progs v5.16.2, so that would need to be upgraded manually.)

https://github.com/kdave/btrfs-progs/issues/522#issuecomment-1973892509

btrfs-progs-5.19.1 (2022-09-12)
"""""""""""""""""""""""""""""""
   * fix memory leaks (extent buffer, path)
   * check: verify block device size vs item <-----
   * rescue fix-device-size: allow to shrink device item <-----
   * receive: fix crash on wrong pinter free()
   * other:
      * experimental: support for block-group-tree
      * documentation updates
      * new tests

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.