btrfs: trimming a btrfs device which has been shrunk previously fails and fills root disk with garbage data

Bug #1896154 reported by Matthew Ruffell
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Matthew Ruffell
linux-azure (Ubuntu)
New
Undecided
Unassigned
Focal
Fix Released
Undecided
Unassigned

Bug Description

BugLink: https://bugs.launchpad.net/bugs/1896154

[Impact]

Since 929be17a9b49 ("btrfs: Switch btrfs_trim_free_extents to find_first_clear_extent_bit") which landed in 5.3, btrfs wont trim a range that has already been trimmed, and will instead go looking for a range where the CHUNK_TRIMMED and CHUNK_ALLOCATED bits aren't set.

If a device had been shrunk, the CHUNK_TRIMMED and CHUNK_ALLOCATED bits are never cleared, which means that btrfs could go looking for a range to trim which is beyond the new device size. This leads to an underflow in a length calculation for the range to trim, and we will end up trimming past the device's boundary.

This has an unfortunate side effect of mangling and filling the root disk with garbage data, and it will not stop until the root disk is totally filled, and makes the instance unusable.

[Fix]

The issue was fixed in the following commit, in 5.9-rc1:

commit c57dd1f2f6a7cd1bb61802344f59ccdc5278c983
Author: Qu Wenruo <email address hidden>
Date: Fri Jul 31 19:29:11 2020 +0800
Subject: btrfs: trim: fix underflow in trim length to prevent access beyond device boundary
Link: https://github.com/torvalds/linux/commit/c57dd1f2f6a7cd1bb61802344f59ccdc5278c983

The fix clears the CHUNK_TRIMMED and CHUNK_ALLOCATED bits when a device is being shrunk, and performs some additional checks to ensure we do not trim past the device size boundary.

The fix was backported to 5.7.17 and 5.8.3 upstream stable, but it seems 5.4 was skipped.

The patch required a minor backport to 5.4, with the CHUNK_STATE_MASK #define moving files back to fs/btrfs/extent_io.h, as the file had been renamed in later kernels.

[Testcase]

The easiest way to reproduce is to use a cloud instance that supplies a real NVMe drive, that supports TRIM and block discards.

Warning, this will fill the root disk with garbage data, ONLY run on a throwaway instance!

Run the following commands:

$ dev=/dev/nvme0n1
$ mnt=/mnt
$ mkfs.btrfs -f $dev -b 10G
$ mount $dev $mnt
$ fstrim $mnt
$ btrfs filesystem resize 1:-1G $mnt
$ fstrim $mnt

The last command will appear to hang, while the root filesystem will begin filling with garbage data. Once the root filesystem fills, you will see the following error:

fstrim: /mnt: FITRIM ioctl failed: Input/output error
/dev/sda1 29G 29G 0 100% /

A test kernel is available from the following PPA:

https://launchpad.net/~mruffell/+archive/ubuntu/sf293389-test

If you install the test kernel, then the final fstrim command completes successfully in a short amount of time.

[Regression Potential]

If a regression were to occur, it could affect users who are attempting to shrink or resize their btrfs volume. Most users already understand that changing the size of a volume is a risky operation, and would have a backup.

If a regression occurs, then there is potential for data loss when users resize or shrink their btrfs volumes. Standard volume creation would not be affected.

The patches have been backported to upstream stable, and are trusted by the community.

Changed in linux (Ubuntu):
status: New → Fix Released
Changed in linux (Ubuntu Focal):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Matthew Ruffell (mruffell)
description: updated
tags: added: sts
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

We are adding this fix earlier to focal:linux-azure since we are preparing a re-spin. focal:linux should receive the fix in the next SRU cycle.

Changed in linux-azure (Ubuntu Focal):
status: New → In Progress
Marcelo Cerri (mhcerri)
Changed in linux-azure (Ubuntu Focal):
status: In Progress → Fix Committed
Ian May (ian-may)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (32.3 KiB)

This bug was fixed in the package linux-azure - 5.4.0-1031.32

---------------
linux-azure (5.4.0-1031.32) focal; urgency=medium

  [ Ubuntu: 5.4.0-51.56 ]

  * Packaging resync (LP: #1786013)
    - update dkms package versions

linux-azure (5.4.0-1030.31) focal; urgency=medium

  [ Ubuntu: 5.4.0-50.55 ]

  * CVE-2020-16119
    - SAUCE: dccp: avoid double free of ccid on child socket
  * CVE-2020-16120
    - Revert "UBUNTU: SAUCE: overlayfs: ensure mounter privileges when reading
      directories"
    - ovl: pass correct flags for opening real directory
    - ovl: switch to mounter creds in readdir
    - ovl: verify permissions in ovl_path_open()
    - ovl: call secutiry hook in ovl_real_ioctl()
    - ovl: check permission to open real file

linux-azure (5.4.0-1029.29) focal; urgency=medium

  * focal/linux-azure: 5.4.0-1029.29 -proposed tracker (LP: #1897102)

  * btrfs: trimming a btrfs device which has been shrunk previously fails and
    fills root disk with garbage data (LP: #1896154)
    - btrfs: trim: fix underflow in trim length to prevent access beyond device
      boundary

  * Mellanox patch for fixing failures in ConnectX3 Pro/DPDK (LP: #1896760)
    - RDMA/mlx4: Read pkey table length instead of hardcoded value

linux-azure (5.4.0-1027.27) focal; urgency=medium

  * focal/linux-azure: 5.4.0-1027.27 -proposed tracker (LP: #1895994)

  * Focal update: v5.4.61 upstream stable release (LP: #1893115)
    - azure: [Config] update config for SPI_DYNAMIC

  * Unexpected warning when hibernating (LP: #1895980)
    - x86/hyperv: Properly suspend/resume reenlightenment notifications

  * [linux-azure] [SRU] UBUNTU: SAUCE: Drivers: hv: vmbus: Add timeout to
    vmbus_wait_for_unload (LP: #1895527)
    - SAUCE: Drivers: hv: vmbus: Add timeout to vmbus_wait_for_unload

  [ Ubuntu: 5.4.0-49.53 ]

  * focal/linux: 5.4.0-49.53 -proposed tracker (LP: #1896007)
  * Comet Lake PCH-H RAID not support on Ubuntu20.04 (LP: #1892288)
    - ahci: Add Intel Comet Lake PCH-H PCI ID
  * Novalink (mkvterm command failure) (LP: #1892546)
    - tty: hvcs: Don't NULL tty->driver_data until hvcs_cleanup()
  * Oops and hang when starting LVM snapshots on 5.4.0-47 (LP: #1894780)
    - SAUCE: Revert "mm: memcg/slab: fix memory leak at non-root kmem_cache
      destroy"
  * Intel x710 LOMs do not work on Focal (LP: #1893956)
    - i40e: Fix LED blinking flow for X710T*L devices
    - i40e: enable X710 support
  * Add/Backport EPYC-v3 and EPYC-Rome CPU model (LP: #1887490)
    - kvm: svm: Update svm_xsaves_supported
  * Fix non-working NVMe after S3 (LP: #1895718)
    - SAUCE: PCI: Enable ACS quirk on CML root port
  * Focal update: v5.4.65 upstream stable release (LP: #1895881)
    - ipv4: Silence suspicious RCU usage warning
    - ipv6: Fix sysctl max for fib_multipath_hash_policy
    - netlabel: fix problems with mapping removal
    - net: usb: dm9601: Add USB ID of Keenetic Plus DSL
    - sctp: not disable bh in the whole sctp_get_port_local()
    - taprio: Fix using wrong queues in gate mask
    - tipc: fix shutdown() of connectionless socket
    - net: disable netpoll on fresh napis
    - Linux 5.4.65
  * Focal update: v5.4.64 upstream stable release (LP: #1895...

Changed in linux-azure (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Matthew Ruffell (mruffell) wrote :

Performing verification for Focal.

I created a i3.large instance on AWS, since it has 1x NVMe drive that supports trim and block discard.

I ensured that I could reproduce the problem with 5.4.0-54-generic from -updates, and I followed the instructions in the Testcase section, and the final fstrim after shrinking locked up the instance, and filled up the root disk. I terminated the instance.

I then created a new instance, and enabled -proposed, and installed 5.4.0-55-generic, and rebooted. From there, I ran though the test steps again:

$ uname -rv
5.4.0-55-generic #61-Ubuntu SMP Mon Nov 9 20:49:56 UTC 2020
$ sudo -s
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 28.1M 1 loop /snap/amazon-ssm-agent/2012
loop1 7:1 0 97.8M 1 loop /snap/core/10185
loop2 7:2 0 55.3M 1 loop /snap/core18/1885
loop3 7:3 0 70.6M 1 loop /snap/lxd/16922
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
nvme0n1 259:0 0 442.4G 0 disk
# dev=/dev/nvme0n1
# mnt=/mnt
# mkfs.btrfs -f $dev -b 10G
btrfs-progs v5.4.1
See http://btrfs.wiki.kernel.org for more information.

Detected a SSD, turning off metadata duplication. Mkfs with -m dup if you want to force metadata duplication.
Label: (null)
UUID: db9dd9f5-7993-4827-9a43-93a72a73aa3a
Node size: 16384
Sector size: 4096
Filesystem size: 10.00GiB
Block group profiles:
  Data: single 8.00MiB
  Metadata: single 8.00MiB
  System: single 4.00MiB
SSD detected: yes
Incompat features: extref, skinny-metadata
Checksum: crc32c
Number of devices: 1
Devices:
   ID SIZE PATH
    1 10.00GiB /dev/nvme0n1

# mount $dev $mnt
# fstrim $mnt
# btrfs filesystem resize 1:-1G $mnt
Resize '/mnt' of '1:-1G'
# fstrim $mnt
#

The final fstrim completed almost immediately, the same speed as the initial fstrim. The instance did not lock up, and the root disk did not get filled with any garbage data.

The kernel in -proposed fixes the problem, happy to mark as verified.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (78.9 KiB)

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.