xfs/073 test fails with Metadata corruption detected on xfs file system (xfsprogs)

[bugproxy]

Impact
======
When making multiple copies of a V5 filesystem with xfs_copy those copies will be corrupt due to incorrect UUIDs.

Test Case
=========
See the detailed steps to create the test environment in comment #20.

Regression Potential
====================
From the patch "Most of this patch is changing comments and re-ordering tests to match; the functional change is to simply use the *sb rather than the *ag_hdr to identify the proper metadata UUID." So this seems pretty straightforward and given that copy process is currently broken, because the copies are corrupt, this will be an improvement.

Original Description
====================

Problem Description
--------------------------------
xfs/073 test fails with Metadata corruption detected on xfs file system. Test fails with _check_xfs_filesystem: filesystem on /mnt/test/84004.image2 is inconsistent.

# diff -u tests/xfs/073.out /root/xfstests-dev/results//xfs/073.out.bad
--- tests/xfs/073.out 2017-03-23 12:13:05.288877197 +0530
+++ /root/xfstests-dev/results//xfs/073.out.bad 2017-03-27 11:11:43.023059702 +0530
@@ -59,8 +59,7 @@
 comparing new image geometry to old
 unmounting and removing new image
 checking new image
-mounting new image on loopback
-comparing new image files to old
-comparing new image directories to old
-comparing new image geometry to old
-unmounting and removing new image
+_check_xfs_filesystem: filesystem on /mnt/test/15413.image2 is inconsistent (c)
+(see /root/xfstests-dev/results//xfs/073.full for details)
+_check_xfs_filesystem: filesystem on /mnt/test/15413.image2 is inconsistent (r)
+(see /root/xfstests-dev/results//xfs/073.full for details)

Metadata corruption detected at xfs_agf block 0x1/0x200

# uname -a
Linux ltc-tuleta12 4.10.0-21-generic #23~16.04.1-Ubuntu SMP Tue May 2 12:54:57 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

Steps to reproduce:
----------------------------------------
1. Create a loop device with xfs filesystem
2. git clone git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git; cd xfstests-dev
3. make
4. Create a local.conf for running with created loop device
5.. Run xfstests-dev test : ./check tests/xfs/073

Full log is attached.

== Comment: #2 - Harish Sriram <email address hidden> - 2017-05-31 01:22:11 ==
(In reply to comment #1)
> Hi Harish,
> Can you share the steps used in creating the loop device with xfs filesystem
> ?
>
> Thank you.

Create loop device:
# mkdir /mnt/loop-device /mnt/test /mnt/scratch

# for i in $(seq 0 1); do fallocate -o 0 -l 5GiB /mnt/loop-device/file-$i.img; done
# for i in $(seq 0 1); do losetup /dev/loop$i /mnt/loop-device/file-$i.img; done

Create File system:
# for i in $(seq 0 1); do mkfs.ext4 -F /dev/loop$i; done

# cat local.config
export TEST_DEV=/dev/loop0
export TEST_DIR=/mnt/test
export SCRATCH_DEV=/dev/loop1
export SCRATCH_MNT=/mnt/scratch

[bugproxy (bugproxy)]

Default Comment by Bridge

[Christian Ehrhardt  (paelzer)]

It should not be mkfs.ext4 I think, but with the slightly adapted following sequence it worked to reproduce:

# mkdir /mnt/loop-device /mnt/test /mnt/scratch
# for i in $(seq 0 1); do fallocate -o 0 -l 5GiB /mnt/loop-device/file-$i.img; done
# for i in $(seq 0 1); do losetup /dev/loop$i /mnt/loop-device/file-$i.img; done
# if yours are busy use --find --show and adapt config
Create File system:
# for i in $(seq 0 1); do mkfs.xfs -F /dev/loop$i; done

# git clone http://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
# apt install libacl1-dev
# make

# cat local.config
export TEST_DEV=/dev/loop0
export TEST_DIR=/mnt/test
export SCRATCH_DEV=/dev/loop1
export SCRATCH_MNT=/mnt/scratch

[Christian Ehrhardt  (paelzer)]

I wonder if that should at least hold an additional task for the kernel - since I'm not sure if this is kernel or xfsutils

[Christian Ehrhardt  (paelzer)]

Oh btw reproduced on x86

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc4

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-07-14 06:04 EDT-------
Tested with 4.12.1 from given link on Ubuntu 17.10

# uname -a
Linux ltc-test-ci2 4.12.1-041201-generic #201707121132 SMP Wed Jul 12 17:03:25 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

Issue is still observed.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-07-19 09:06 EDT-------
This xfsprogs commit cbca895541facb3a1a00fd0fe6614301a64c0e3a fixes the issue.
Built the deb file and tested in 17.10. Issue is not observed.

Patch file will be attached.

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2017-07-19 09:10 EDT-------

[Steve Langasek (vorlon)]

The attached patch is taken from xfsprogs upstream and is present in versions 4.8.0 and later. Artful is currently behind Debian, which has 4.9.0 (Artful has 4.3).

[Launchpad Janitor (janitor)]

This bug was fixed in the package xfsprogs - 4.9.0+nmu1ubuntu1

---------------
xfsprogs (4.9.0+nmu1ubuntu1) artful; urgency=low

  * Merge from Debian unstable.
    - Fixes a metadata corruption bug. LP: #1696102.
  * Remaining changes:
    - Enable blkid in d-i, by dropping explicitly passed configure option
      (broken configure).
    - Drop trying to create upstream distribution.
  * Dropped changes, included in Debian:
    - Add autotools check for struct fsxattr to fix FTBFS.
    - Use FIFREEZE and FITHAW when defined.
    - 2be9e939868400 "linux.h: include <linux/fs.h>"
    - 721948f7688e7d "xfs.h: define XFS_IOC_FREEZE even if FIFREEZE is defined"
    - 83f4b5ac15fa3d "xfs_fs.h: XFS_IOC_FS[SG]SETXATTR to FS_IOC_FS[SG]ETXATTR
      promotion"

xfsprogs (4.9.0+nmu1) unstable; urgency=medium

  * Non-maintainer upload.
  * Remove code in include/buildmacros which creates bogus symlinks for
    libraries. (Closes: #766811)

xfsprogs (4.9.0) unstable; urgency=low

  * New upstream release

xfsprogs (4.8.0) unstable; urgency=low

  * New upstream release

xfsprogs (4.7.0) unstable; urgency=low

  * New upstream release

xfsprogs (4.5.0) unstable; urgency=low

  * New upstream release

 -- Steve Langasek <email address hidden> Tue, 25 Jul 2017 22:33:17 -0700

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2017-07-19 09:10 EDT-------

[Brian Murray (brian-murray)]

I was looking at backporting this fix to Ubuntu 17.04 and Ubuntu 16.04 but given that the test is somewhat complicated I first decided to run the tests against Ubuntu 17.10 (Artful). While doing so I discovered that the test tests/xfs/073 still fails for me with xfsprogs version 4.9.0+nmu1ubuntu1. Could somebody else please test that version of xfsprogs and see if the test passes? Thanks!

 $ sudo ./check tests/xfs/073
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/x86_64 impulse 4.12.0-11-generic
MKFS_OPTIONS -- -f -bsize=4096 /dev/loop9
MOUNT_OPTIONS -- /dev/loop9 /mnt/scratch

xfs/073 - output mismatch (see /tmp/xfstests-dev/results//xfs/073.out.bad)
    --- tests/xfs/073.out 2017-08-29 12:01:20.967709921 -0700
    +++ /tmp/xfstests-dev/results//xfs/073.out.bad 2017-08-29 13:54:58.307799243 -0700
    @@ -44,6 +44,14 @@
     comparing new image files to old
     comparing new image directories to old
     comparing new image geometry to old
    +--- /tmp/24787.geometry1 2017-08-29 13:54:57.419799231 -0700
    ++++ /tmp/24787.geometry2 2017-08-29 13:54:57.419799231 -0700
    +@@ -1,4 +1,4 @@
    +-meta-data=<DEVIMAGE> isize=512 agcount=4, agsize=6553600 blks
    ...
    (Run 'diff -u tests/xfs/073.out /tmp/xfstests-dev/results//xfs/073.out.bad' to see the entire diff)
Ran: xfs/073
Failures: xfs/073
Failed 1 of 1 tests

[Steve Langasek (vorlon)]

> (Run 'diff -u tests/xfs/073.out /tmp/xfstests-dev/results//xfs/073.out.bad' to see the entire diff)

Can you show the full output of this diff?

[Brian Murray (brian-murray)]

 $ diff -u tests/xfs/073.out /tmp/xfstests-dev/results//xfs/073.out.bad
--- tests/xfs/073.out 2017-08-29 12:01:20.967709921 -0700
+++ /tmp/xfstests-dev/results//xfs/073.out.bad 2017-08-29 13:54:58.307799243 -0700
@@ -44,6 +44,14 @@
 comparing new image files to old
 comparing new image directories to old
 comparing new image geometry to old
+--- /tmp/24787.geometry1 2017-08-29 13:54:57.419799231 -0700
++++ /tmp/24787.geometry2 2017-08-29 13:54:57.419799231 -0700
+@@ -1,4 +1,4 @@
+-meta-data=<DEVIMAGE> isize=512 agcount=4, agsize=6553600 blks
++meta-data=<DEVIMAGE>0 isize=512 agcount=4, agsize=6553600 blks
+ = sectsz=512 attr=2, projid32bit=1
+ = crc=1 finobt=1 spinodes=0 rmapbt=0
+ = reflink=0
 unmounting and removing new image

 === copying scratch device to multiple targets

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-08-30 01:17 EDT-------
This issue is fixed on ppc64le architecture with xfsprogs-4.9.0+nmu1ubuntu1 on artful.

# ./check tests/xfs/073
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/ppc64le pkvmhab010 4.12.0-11-generic
MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1
MOUNT_OPTIONS -- /dev/loop1 /mnt/scratch

xfs/073 11s
Ran: xfs/073
Passed all 1 tests

Thanks,
Harish

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2017-07-19 09:10 EDT-------

[Christian Ehrhardt  (paelzer)]

I was asked to also recheck:
Working on artful for me as well:

<get the tests to build>
<setup as described 2x2G is enough>

[...]
root@cpaelzer-artful:/home/ubuntu/xfstests-dev# ./check tests/xfs/073
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/ppc64le cpaelzer-artful 4.11.0-10-generic
MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1
MOUNT_OPTIONS -- /dev/loop1 /mnt/scratch

xfs/073 4s
Ran: xfs/073
Passed all 1 tests

xfsprogs 4.9.0+nmu1ubuntu1 ppc64el

---

Comparing Xenial:
./check tests/xfs/073
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/ppc64le cpaelzer-xenial 4.4.0-92-generic
MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1
MOUNT_OPTIONS -- /dev/loop1 /mnt/scratch

xfs/073 [failed, exit status 1] - output mismatch (see /home/ubuntu/xfstests-dev/results//xfs/073.out.bad)
    --- tests/xfs/073.out 2017-08-31 08:53:15.888003364 +0000
    +++ /home/ubuntu/xfstests-dev/results//xfs/073.out.bad 2017-08-31 08:59:47.111628779 +0000
    @@ -59,8 +59,7 @@
     comparing new image geometry to old
     unmounting and removing new image
     checking new image
    -mounting new image on loopback
    -comparing new image files to old
    -comparing new image directories to old
    -comparing new image geometry to old
    ...
    (Run 'diff -u tests/xfs/073.out /home/ubuntu/xfstests-dev/results//xfs/073.out.bad' to see the entire diff)
Ran: xfs/073
Failures: xfs/073
Failed 1 of 1 tests

xfsprogs 4.3.0+nmu1ubuntu1

---

@bdmurray - I left you two KVM guests that are set up with that on Diamond and imported your keys there, ping me if you need help to access (but vorlon can help as well if I'm out of reach).
- cpaelzer-artful
- cpaelzer-xenial

[Brian Murray (brian-murray)]

I've uploaded this to the xenial and zesty -proposed queues for review by the SRU team, however the test case / steps to reproduce could use some improvement see bug 1704932 for a better example.

[Łukasz Zemczak (sil2100)]

Since I don't have much experience in the xfsprogs code, I would also appreciate some Regression Potential section in the description before fully approving the upload.

[Christian Ehrhardt  (paelzer)]

[21:30] <bdmurray> Could you update the test case with the setup you configured on diamond? I just used the environment you had configured so I'm not convinced what I tried at home was good.

Helping where I can :-)
Trying to add a full path of "how to reproduce"

# Prep VM
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
$ uvt-kvm create --password=ubuntu artful-test-xfs release=xenial arch=amd64 label=daily
$ uvt-kvm ssh --insecure xenial-test-xfs
# all following in the VM

# Prep (fake) Devices
$ sudo apt update
$ sudo apt install xfsprogs
$ sudo mkdir /mnt/loop-device /mnt/test /mnt/scratch
$ for i in $(seq 0 1); do sudo fallocate -o 0 -l 1GiB /mnt/loop-device/file-$i.img; done
$ for i in $(seq 0 1); do sudo losetup /dev/loop$i /mnt/loop-device/file-$i.img; done
$ for i in $(seq 0 1); do sudo mkfs.xfs /dev/loop$i; done

# make the test build
$ git clone http://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
$ cd xfstests-dev/
$ sudo apt-get install bc xfslibs-dev uuid-dev libtool-bin e2fsprogs automake gcc libuuid1 quota attr libattr1-dev make libacl1-dev libaio-dev xfsprogs libgdbm-dev gawk fio dbench uuid-runtime
$ make

# Configure and run the test
$ cat << EOF > local.config
export TEST_DEV=/dev/loop0
export TEST_DIR=/mnt/test
export SCRATCH_DEV=/dev/loop1
export SCRATCH_MNT=/mnt/scratch
EOF
$ sudo ./check tests/xfs/073

Output:
FSTYP -- xfs (non-debug)
PLATFORM -- Linux/x86_64 xenial-xfs-test 4.4.0-93-generic
MKFS_OPTIONS -- -f -bsize=4096 /dev/loop1
MOUNT_OPTIONS -- /dev/loop1 /mnt/scratch

xfs/073 [failed, exit status 1] - output mismatch (see /home/ubuntu/xfstests-dev/results//xfs/073.out.bad)
    --- tests/xfs/073.out 2017-09-08 07:09:40.754427994 +0000
    +++ /home/ubuntu/xfstests-dev/results//xfs/073.out.bad 2017-09-08 07:12:08.382624013 +0000
    @@ -59,8 +59,7 @@
     comparing new image geometry to old
     unmounting and removing new image
     checking new image
    -mounting new image on loopback
    -comparing new image files to old
    -comparing new image directories to old
    -comparing new image geometry to old
    ...
    (Run 'diff -u tests/xfs/073.out /home/ubuntu/xfstests-dev/results//xfs/073.out.bad' to see the entire diff)
Ran: xfs/073
Failures: xfs/073
Failed 1 of 1 tests

[Christian Ehrhardt  (paelzer)]

And @bdmurray - what you tried surely was good - don't be overmodest :-)

Never the less exact step-by-step commands can't hurt.

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted xfsprogs into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xfsprogs/4.3.0+nmu1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello bugproxy, or anyone else affected,

Accepted xfsprogs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/xfsprogs/4.3.0+nmu1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-09-22 13:12 EDT-------
(In reply to comment #21)
> Accepted xfsprogs into zesty-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/xfsprogs/4.3.0+nmu1ubuntu4.1 in a few
> hours, and then in the -proposed repository.

The issue is resolved on zesty

xfs/073 12s
Ran: xfs/073
Passed all 1 tests

# dpkg -l | grep xfsp
ii xfsprogs 4.3.0+nmu1ubuntu4.1 ppc64el Utilities for managing the XFS filesystem

> Accepted xfsprogs into xenial-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/xfsprogs/4.3.0+nmu1ubuntu1.1 in a few
> hours, and then in the -proposed repository.
>

The issue is resolved on xenial

xfs/073 15s
Ran: xfs/073
Passed all 1 tests

# dpkg -l | grep xfsp
ii xfsprogs 4.3.0+nmu1ubuntu1.1 ppc64el Utilities for managing the XFS filesystem

Thanks,
Harish

[Launchpad Janitor (janitor)]

This bug was fixed in the package xfsprogs - 4.3.0+nmu1ubuntu1.1

---------------
xfsprogs (4.3.0+nmu1ubuntu1.1) xenial; urgency=medium

  * xfs_copy: Fix meta UUID handling on multiple copies upstream fix
    cbca895541facb. (LP: #1696102)

 -- Brian Murray <email address hidden> Tue, 05 Sep 2017 09:59:09 -0700

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for xfsprogs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package xfsprogs - 4.3.0+nmu1ubuntu4.1

---------------
xfsprogs (4.3.0+nmu1ubuntu4.1) zesty; urgency=medium

  * xfs_copy: Fix meta UUID handling on multiple copies upstream fix
    cbca895541facb. (LP: #1696102)

 -- Brian Murray <email address hidden> Tue, 29 Aug 2017 10:16:48 -0700