zfs diff: Unable to determine path or stats for object

Bug #1849665 reported by Andreas Hasenack on 2019-10-24
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
High
Colin Ian King
Eoan
Undecided
Unassigned

Bug Description

== SRU Justification, Eoan ==

Using zfs diff on an encrypted dataset with large objects one can hit an error such as follows:

# zfs diff nsnx/trusty-2a@snap1 nsnx/trusty-2a
+ /nsnx/trusty-2a/bin
Unable to determine path or stats for object 5 in nsnx/trusty-2a@zfs-diff-32359-000000010001f165: File exists

== Fix ==

Upstream commit d359e99c38f667 ("diff_cb() does not handle large dnodes") as addressed in ZFS bug fix: https://github.com/zfsonlinux/zfs/pull/9343

== Testcase ==

# mkdir /zfs-test
# cd /zfs-test
# truncate -s 10G file.img
# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,304365 s, 138 MB/s
# zfs snapshot tank/d1@s1
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,305324 s, 137 MB/s

Without the fix, one hits an error such as:

# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists

With the fix, we get:
+ /tank/d1/somedata2.bin
M /tank/d1/

== Regression Potential ==

This is a minor change in module/zfs/dmu_diff.c and it only affects the zfs diff component, so this should not affect ZFS in terms of file system corruption/data loss. This has also been upstream regression tested and passes the Ubuntu ZFS regressions tests too. So the risk is limited.

---------------------

Eoan 19.10
zfsutils-linux 0.8.1-1ubuntu14
kernel 5.3.0-19-generic #20-Ubuntu

When using zfs diff on an encrypted dataset, I frequently encounter this error:

# zfs diff nsnx/trusty-2a@snap1 nsnx/trusty-2a
+ /nsnx/trusty-2a/bin
Unable to determine path or stats for object 5 in nsnx/trusty-2a@zfs-diff-32359-000000010001f165: File exists

I believe this to be upstream bug https://github.com/zfsonlinux/zfs/issues/7678, fixed with https://github.com/zfsonlinux/zfs/pull/9343

Here is one way to reproduce it:

# mkdir /zfs-test
# cd /zfs-test
# truncate -s 10G file.img
# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,304365 s, 138 MB/s
# zfs snapshot tank/d1@s1
# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,305324 s, 137 MB/s
# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists

There may be a simpler way to test this, but this should be enough to start with.

description: updated
Changed in zfs-linux (Ubuntu):
importance: Undecided → High
assignee: nobody → Colin Ian King (colin-king)
status: New → In Progress
Colin Ian King (colin-king) wrote :
Colin Ian King (colin-king) wrote :

With the fix:

root@eoan-amd64-efi:/home/cking# mkdir /zfs-test
root@eoan-amd64-efi:/home/cking# cd /zfs-test
root@eoan-amd64-efi:/zfs-test# truncate -s 10G file.img
root@eoan-amd64-efi:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@eoan-amd64-efi:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@eoan-amd64-efi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.238499 s, 176 MB/s
root@eoan-amd64-efi:/zfs-test# zfs snapshot tank/d1@s1
root@eoan-amd64-efi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.228746 s, 183 MB/s
root@eoan-amd64-efi:/zfs-test# zfs diff tank/d1@s1 tank/d1
+ /tank/d1/somedata2.bin
M /tank/d1/

description: updated
Colin Ian King (colin-king) wrote :

I'll get this uploaded into -proposed once the current SRU backlog is out.

Andreas Hasenack (ahasenack) wrote :

Thanks!

Colin Ian King (colin-king) wrote :

uploaded zfs-linux (0.8.1-1ubuntu14.1) eoan (will land in -proposed sometime soon)
uploaded zfs-linux (0.0.1.1ubuntu16) focal

Once the packages are uploaded the dkms driver component will be sync'd into the next kernel and then once this is in -proposed it can be fully tested.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.1-1ubuntu16

---------------
zfs-linux (0.8.1-1ubuntu16) focal; urgency=medium

  * 4600-diff_cb-does-not-handle-large-dnodes.patch
    - fix zfs diff on snapshots with large dnodes (LP: #1849665)
      Minor backport of upstream ZFS commit d359e99c38f667
      ("diff_cb() does not handle large dnodes")

 -- Colin Ian King <email address hidden> Wed, 24 Oct 2019 16:07:23 +0100

Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released

Hello Andreas, or anyone else affected,

Accepted zfs-linux into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.1-1ubuntu14.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Eoan):
status: New → Fix Committed
Andreas Hasenack (ahasenack) wrote :

Eoan verification

First, reproducing the bug:

root@nsn7:~# mkdir /zfs-test
root@nsn7:~# cd /zfs-test/
root@nsn7:/zfs-test# truncate -s 10G file.img
root@nsn7:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@nsn7:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,425336 s, 98,6 MB/s
root@nsn7:/zfs-test# zfs snapshot tank/d1@s1
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,260445 s, 161 MB/s
root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-19-generic #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Now installing the updated kernel and rebooting into it.

Andreas Hasenack (ahasenack) wrote :

Hm, at first glance this didn't work with the new kernel and zfsutils-linux package:

root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-21-generic #22-Ubuntu SMP Tue Oct 29 22:55:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@nsn7:/zfs-test# dpkg -l|grep linux-image
ii linux-image-5.3.0-18-generic 5.3.0-18.19+1 amd64 Signed kernel image generic
ii linux-image-5.3.0-19-generic 5.3.0-19.20 amd64 Signed kernel image generic
ii linux-image-5.3.0-21-generic 5.3.0-21.22 amd64 Signed kernel image generic
ii linux-image-generic 5.3.0.21.24 amd64 Generic Linux kernel image
root@nsn7:/zfs-test#

This was with the existing pool and dataset from the previous test.

Let me try restarting it from scratch:
root@nsn7:/zfs-test# cd
root@nsn7:~# zpool destroy tank
root@nsn7:~# rm -rf /zfs-test/
root@nsn7:~# mkdir /zfs-test
root@nsn7:~# cd /zfs-test
root@nsn7:/zfs-test# truncate -s 10G file.img
root@nsn7:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@nsn7:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,39306 s, 107 MB/s
root@nsn7:/zfs-test# zfs snapshot tank/d1@s1
root@nsn7:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0,265228 s, 158 MB/s
root@nsn7:/zfs-test# zfs diff tank/d1@s1 tank/d1
Unable to determine path or stats for object 3 in tank/d1@s1: File exists
root@nsn7:/zfs-test# uname -a
Linux nsn7 5.3.0-21-generic #22-Ubuntu SMP Tue Oct 29 22:55:51 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

root@nsn7:/zfs-test# apt-cache policy linux-image-5.3.0-21-generic zfsutils-linux
linux-image-5.3.0-21-generic:
  Installed: 5.3.0-21.22
  Candidate: 5.3.0-21.22
  Version table:
 *** 5.3.0-21.22 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
zfsutils-linux:
  Installed: 0.8.1-1ubuntu14.1
  Candidate: 0.8.1-1ubuntu14.1
  Version table:
 *** 0.8.1-1ubuntu14.1 500
        500 http://br.archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.1-1ubuntu14 500
        500 http://br.archive.ubuntu.com/ubuntu eoan/main amd64 Packages

Same thing.

Colin Ian King (colin-king) wrote :

So this is a 2-phase fix. The dkms package is updated, then we test this, then this gets sync'd into the kernel. I'm testing it right now, let me sanity check the zfs-dkms part first and get that updated as step #1.

Colin Ian King (colin-king) wrote :

Installed the new zfsutils + zfs dkms to sanity check the kernel driver part of the fix:

dmesg | grep ZFS
[ 22.420188] ZFS: Loaded module v0.8.1-1ubuntu14.1, ZFS pool version 5000, ZFS filesystem version 5

And now the test:

root@eoan-amd64-uefi:~# mkdir /zfs-test
root@eoan-amd64-uefi:~# cd /zfs-test
root@eoan-amd64-uefi:/zfs-test# truncate -s 10G file.img
root@eoan-amd64-uefi:/zfs-test# zpool create -o ashift=12 -O acltype=posixacl -O compression=lz4 -O xattr=sa -O normalization=formD -O dnodesize=auto tank $(pwd)/file.img
root@eoan-amd64-uefi:/zfs-test# zfs create tank/d1 -o encryption=on -o keyformat=passphrase
Enter passphrase:
Re-enter passphrase:
root@eoan-amd64-uefi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.319657 s, 131 MB/s
root@eoan-amd64-uefi:/zfs-test# zfs snapshot tank/d1@s1
root@eoan-amd64-uefi:/zfs-test# dd if=/dev/urandom bs=4k of=/tank/d1/somedata2.bin count=10240
10240+0 records in
10240+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.312195 s, 134 MB/s
root@eoan-amd64-uefi:/zfs-test# zfs diff tank/d1@s1 tank/d1
M /tank/d1/
+ /tank/d1/somedata2.bin

The zfsutils + dkms package has the fix. Once this lands we can then sync this into the next kernel release for the complete fix.

tags: added: verification-done-eoan

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.1-1ubuntu14.1

---------------
zfs-linux (0.8.1-1ubuntu14.1) eoan; urgency=medium

  * 4600-diff_cb-does-not-handle-large-dnodes.patch
    - fix zfs diff on snapshots with large dnodes (LP: #1849665)
      Minor backport of upstream ZFS commit d359e99c38f667
      ("diff_cb() does not handle large dnodes")

 -- Colin Ian King <email address hidden> Wed, 24 Oct 2019 16:07:23 +0100

Changed in zfs-linux (Ubuntu Eoan):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers