pvmove causes file system corruption without notice upon move from 512 -> 4096 logical block size devices
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on IBM z Systems |
Fix Released
|
Medium
|
Canonical Foundations Team | ||
lvm2 |
Fix Released
|
Medium
|
|||
e2fsprogs (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
linux (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
lvm2 (Ubuntu) |
Invalid
|
Undecided
|
Skipper Bug Screeners |
Bug Description
Problem Description---
Summary
=======
Environment: IBM Z13 LPAR and z/VM Guest
IBM Type: 2964 Model: 701 NC9
OS: Ubuntu 18.10 (GNU/Linux 4.18.0-13-generic s390x)
LVM: pvmove operation corrupts file system when using 4096 (4k) logical block size
and default block size being 512 bytes in the underlying devices
The problem is immediately reproducible.
We see a real usability issue with data destruction as consequence - which is not acceptable.
We expect 'pvmove' to fail with error in such situations to prevent fs destruction,
which might possibly be overridden by a force flag.
Details
=======
After a 'pvmove' operation is run to move a physical volume onto an ecrypted
device with 4096 bytes logical block size we experience a file system corruption.
There is no need for the file system to be mounted, but the problem surfaces
differently if so.
Either, the 'pvs' command after the pvmove shows
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument
/dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument
or
a subsequent mount shows (after umount if the fs had previously been mounted as in our
setup)
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/mapper/
A minimal setup of LVM using one volume group with one logical volume defined,
based on one physical volume is sufficient to raise the problem. One more physical
volume of the same size is needed to run the pvmove operation to.
LV
|
VG: LOOP_VG [ ]
|
PV: /dev/loop0 --> /dev/mapper/
The physical volumes are backed by loopback devices (losetup) to base the
problem report on, but we have seen the error on real SCSI multipath volumes
also, with and without cryptsetup mapper devices in use.
Further discussion
==================
https:/
The problem does not occur on block devices with native size of 4k.
E.g. DASDs, or file systems with mkfs -b 4096 option.
Terminal output
===============
See attached file pvmove-error.txt
Debug data
==========
pvmove was run with -dddddd (maximum debug level)
See attached journal file.
Contact Information = <email address hidden>
---uname output---
Linux system 4.18.0-13-generic #14-Ubuntu SMP Wed Dec 5 09:00:35 UTC 2018 s390x s390x s390x GNU/Linux
Machine Type = IBM Type: 2964 Model: 701 NC9
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1.) Create two image files of 500MB in size
and set up two loopback devices with 'losetup -fP FILE'
2.) Create one physical volume and one volume group 'LOOP_VG',
and one logical volume 'VG'
Run:
pvcreate /dev/loop0
vgcreate LOOP_VG /dev/loop0
lvcreate -L 300MB LOOP_VG -n LV /dev/loop0
3.) Create a file system on the logical volume device:
mkfs.ext4 /dev/mapper/
4.) mount the file system created in the previous step to some empty available directory:
mount /dev/mapper/
5.) Set up a second physical volume, this time encrypted with LUKS2,
and open the volume to make it available:
cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1
cryptsetup luksOpen /dev/loop1 enc-loop
6.) Create the second physical volume, and add it to the LOOP_VG
pvcreate /dev/mapper/
vgextend LOOP_VG /dev/mapper/
7.) Ensure the new physical volume is part of the volume group:
pvs
8.) Move the /dev/loop0 volume onto the encrypted volume with maximum debug option:
pvmove -dddddd /dev/loop0 /dev/mapper/
9.) The previous step succeeds, but corrupts the file system on the logical volume
We expect an error here.
There might be a command line flag to override used because corruption does not cause a data loss.
Userspace tool common name: pvmove
The userspace tool has the following bit modes: 64bit
Userspace rpm: lvm2 in versoin 2.02.176-4.1ubuntu3
Userspace tool obtained from project website: na
*Additional Instructions for <email address hidden>:
-Attach ltrace and strace of userspace application.
affects: | linux (Ubuntu) → lvm2 (Ubuntu) |
Changed in ubuntu-z-systems: | |
assignee: | nobody → Canonical Foundations Team (canonical-foundations) |
importance: | Undecided → Critical |
Changed in lvm2: | |
importance: | Unknown → Medium |
status: | Unknown → Confirmed |
Changed in lvm2 (Ubuntu): | |
status: | Incomplete → Invalid |
Changed in e2fsprogs (Ubuntu): | |
status: | New → Fix Committed |
Changed in lvm2: | |
status: | Confirmed → Fix Released |
Description of problem:
lvm should not allow extending an LV with a PV of different sector size than existing PVs making up the LV, since the FS on the LV does not mount once LVM adds in the new PV and extends the LV.
How reproducible:
Steps to Reproduce:
** Device: sdc (using the device with default sector size of 512)
# blockdev --report /dev/sdc
RO RA SSZ BSZ StartSec Size Device
rw 8192 512 4096 0 1073741824 /dev/sdc
** LVM is created with the default sector size of 512.
# blockdev --report /dev/mapper/ testvg- testlv testvg- testlv
RO RA SSZ BSZ StartSec Size Device
rw 8192 512 4096 0 1069547520 /dev/mapper/
** The filesystem will also pick up 512 sector size.
# mkfs.xfs /dev/mapper/ testvg- testlv /dev/mapper/ testvg- testlv isize=512 agcount=4, agsize=65280 blks
meta-data=
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=261120, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
** Now we will mount it
# xfs_info /test /dev/mapper/ testvg- testlv isize=512 agcount=4, agsize=65280 blks
meta-data=
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0 spinodes=0
data = bsize=4096 blocks=261120, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
** Let's extend it with a PV with a sector size of 4096:
#modprobe scsi_debug sector_size=4096 dev_size_mb=512
# fdisk -l /dev/sdd
Disk /dev/sdd: 536 MB, 536870912 bytes, 131072 sectors
Units = sectors of 1 * 4096 = 4096 bytes <==============
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 262144 bytes
# blockdev --report /dev/sdd
RO RA SSZ BSZ StartSec Size Device
rw 8192 4096 4096 0 536870912 /dev/sdd
# vgextend testvg /dev/sdd
Physical volume "/dev/sdd" successfully created
Volume group "testvg" successfully extended
# lvextend -l +100%FREE /dev/mapper/ testvg- testlv
Size of logical volume testvg/testlv changed from 1020.00 MiB (255 extents) to 1.49 GiB (382 extents).
Logical volume testlv successfully resized.
# umount /test
# mount /dev/mapper/ testvg- testlv /test testvg- testlv on /test failed: Function not implemented <===========
mount: mount /dev/mapper/
# dmesg | grep -i dm-2
[ 477.517515] XFS (dm-2): Unmounting Filesystem
[ 486.905933] XFS (dm-2): device supports 4096 byte sectors (n...