raid10: discard leads to corrupted file system
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Trusty |
Invalid
|
Undecided
|
Unassigned | ||
Xenial |
Invalid
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
Unassigned | ||
Focal |
Fix Released
|
High
|
Unassigned | ||
Groovy |
Fix Released
|
High
|
Unassigned |
Bug Description
Seems to be closely related to https:/
After updating the Ubuntu 18.04 kernel from 4.15.0-124 to 4.15.0-126 the fstrim command triggered by fstrim.timer causes a severe number of mismatches between two RAID10 component devices.
This bug affects several machines in our company with different HW configurations (All using ECC RAM). Both, NVMe and SATA SSDs are affected.
How to reproduce:
- Create a RAID10 LVM and filesystem on two SSDs
mdadm -C -v -l10 -n2 -N "lv-raid" -R /dev/md0 /dev/nvme0n1p2 /dev/nvme1n1p2
pvcreate -ff -y /dev/md0
vgcreate -f -y VolGroup /dev/md0
lvcreate -n root -L 100G -ay -y VolGroup
mkfs.ext4 /dev/VolGroup/root
mount /dev/VolGroup/root /mnt
- Write some data, sync and delete it
dd if=/dev/zero of=/mnt/data.raw bs=4K count=1M
sync
rm /mnt/data.raw
- Check the RAID device
echo check >/sys/block/
- After finishing (see /proc/mdstat), check the mismatch_cnt (should be 0):
cat /sys/block/
- Trigger the bug
fstrim /mnt
- Re-Check the RAID device
echo check >/sys/block/
- After finishing (see /proc/mdstat), check the mismatch_cnt (probably in the range of N*10000):
cat /sys/block/
After investigating this issue on several machines it *seems* that the first drive does the trim correctly while the second one goes wild. At least the number and severity of errors found by a USB stick live session fsck.ext4 suggests this.
To perform the single drive evaluation the RAID10 was started using a single drive at once:
mdadm --assemble /dev/md127 /dev/nvme0n1p2
mdadm --run /dev/md127
fsck.ext4 -n -f /dev/VolGroup/root
vgchange -a n /dev/VolGroup
mdadm --stop /dev/md127
mdadm --assemble /dev/md127 /dev/nvme1n1p2
mdadm --run /dev/md127
fsck.ext4 -n -f /dev/VolGroup/root
When starting these fscks without -n, on the first device it seems the directory structure is OK while on the second device there is only the lost+found folder left.
Side-note: Another machine using HWE kernel 5.4.0-56 (after using -53 before) seems to have a quite similar issue.
Unfortunately the risk/regression assessment in the aforementioned bug is not complete: the workaround only mitigates the issues during FS creation. This bug on the other hand is triggered by a weekly service (fstrim) causing severe file system corruption.
tags: | added: sts |
Changed in linux (Ubuntu Bionic): | |
status: | New → In Progress |
Changed in linux (Ubuntu Focal): | |
status: | New → In Progress |
Changed in linux (Ubuntu Groovy): | |
status: | New → In Progress |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Focal): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Groovy): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Bionic): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Groovy): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Groovy): | |
assignee: | nobody → Sinclair Willis (yousure122244444444) |
Changed in linux (Ubuntu Groovy): | |
assignee: | Sinclair Willis (yousure122244444444) → nobody |
Status changed to 'Confirmed' because the bug affects multiple users.