Btrfs: Disk replacement causes massive allocation of empty single chunks while degraded

Bug #1925284 reported by Jonah Sabean
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

I experience this with Ubuntu 21.04. I've not tested on older releases. I have a btrfs RAID1 array and simulated a failed disk and then replaced it with the `btrfs replace` feature part of btrfs-progs after mounting the remaining disk with the degraded,noatime mount options.

For clarity: The disk that is being used in btrfs replace is sda1, sdb1 is the existing disk with the data that is mounted degraded.

Unfortunately, when you begin the replacement, btrfs begins slowly allocating new empty single chunks on the disk that was mounted degraded (sdb1) as soon as the btrfs replace begins, seemingly equal to the amount of raid1 chunks that were already allocated + 1. As the replace continues, this value increases:

Data,single: Size:107.00GiB, Used:5.25MiB (0.00%)
   /dev/sdb1 107.00GiB

... several minutes later as btrfs replace continues to run ...

Data,single: Size:177.00GiB, Used:5.25MiB (0.00%)
   /dev/sdb1 177.00GiB

I have also posted this on the Btrfs kernel mailing list here for more details and output from commands: https://lore.kernel.org<email address hidden>/T/#t

It's speculated this could be an Ubuntu specific issue. What is certain is I've done this process three times with this array on Ubuntu 21.04 and it happens every time, so I can reproduce it 100%. I believe this is a bug with the Ubuntu kernel, but I suppose it could be related to btrfs-progs. What is certain is the btrfs replace triggers the issue.

I'm happy to provide any more additional details as best I can. Thanks,
-Jonah
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu65
Architecture: amd64
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 21.04
InstallationDate: Installed on 2021-04-18 (2 days ago)
InstallationMedia: Kubuntu 21.04 "Hirsute Hippo" - Beta amd64 (20210418)
MachineType: To Be Filled By O.E.M. To Be Filled By O.E.M.
Package: linux (not installed)
ProcEnviron:
 LANGUAGE=en_CA:en
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
ProcFB: 0 amdgpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-14-generic root=UUID=b34e7cba-e2cd-440e-ac70-479267e84867 ro quiet splash vt.handoff=7
ProcVersionSignature: Ubuntu 5.11.0-14.15-generic 5.11.12
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RebootRequiredPkgs:
 linux-image-5.11.0-16-generic
 linux-base
RelatedPackageVersions:
 linux-restricted-modules-5.11.0-14-generic N/A
 linux-backports-modules-5.11.0-14-generic N/A
 linux-firmware 1.197
RfKill:

Tags: hirsute
Uname: Linux 5.11.0-14-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 06/04/2020
dmi.bios.release: 5.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: P3.40
dmi.board.name: B450 Steel Legend
dmi.board.vendor: ASRock
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrP3.40:bd06/04/2020:br5.14:svnToBeFilledByO.E.M.:pnToBeFilledByO.E.M.:pvrToBeFilledByO.E.M.:rvnASRock:rnB450SteelLegend:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.family: To Be Filled By O.E.M.
dmi.product.name: To Be Filled By O.E.M.
dmi.product.sku: To Be Filled By O.E.M.
dmi.product.version: To Be Filled By O.E.M.
dmi.sys.vendor: To Be Filled By O.E.M.

Jonah Sabean (jseyo)
description: updated
description: updated
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1925284

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Jonah Sabean (jseyo) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected hirsute
description: updated
Revision history for this message
Jonah Sabean (jseyo) wrote : AudioDevicesInUse.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : CRDA.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : IwConfig.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : Lspci.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : Lspci-vt.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : Lsusb.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : Lsusb-t.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : Lsusb-v.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : PaInfo.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : ProcModules.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : UdevDb.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : WifiSyslog.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote : acpidump.txt

apport information

Revision history for this message
Jonah Sabean (jseyo) wrote :

Wow, I did not realise apport submitted so many messages.

Anyway, the system wanted a reboot for a kernel update so I proceeded to reboot after making sure I did a `sudo apt update && sudo apt upgrade` to ensure everything is up to date as of this comment.

I've then went ahead to run the replace again, and the behaviour still exists.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Seth Forshee (sforshee) wrote :

I just confirmed that Ubuntu 21.04 has no kernel patches for btrfs that are not from upstream stable releases, so I have doubts that this is Ubuntu-specific. We do produce testing builds of unpatched upstream kernel releases though, so you could try out 5.11 and 5.12 builds to check whether this is an upstream issue. Those builds can be found here:

https://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D

Revision history for this message
Jonah Sabean (jseyo) wrote :

Hello there,
Thanks for the suggestion. I'm beginning to have the same opinion now, this is odd behaviour that I've been able to reproduce on Debian Bullseye (kernel 5.10) as well, both with a freshly formatted filesystem and the same one -- 100% reproducible for each.

I'll proceed to do some more testing, but I believe this is upstream. It seems to not be too bad, as a balance after a replacement can solve the issue, but it's crucial people check `btrfs filesystem usage` after a replace.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.