Fix zfs_get_data access to files with wrong generation causing panics

Bug #1946686 reported by Andrew Berry
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zfs-linux (Ubuntu)
Fix Released
High
Dimitri John Ledkov
Focal
Fix Released
High
Dimitri John Ledkov
Hirsute
Fix Released
High
Dimitri John Ledkov
Impish
Fix Released
High
Dimitri John Ledkov

Bug Description

== SRU Focal/Hirsute ==

[Impact]

ZFS: Fix a panic while acquiring a lock.

If TX_WRITE is create on a file, and the file is later deleted and a new
directory is created on the same object id, it is possible that when
zil_commit happens, zfs_get_data will be called on the new directory.
This may result in panic as it tries to do range lock.

Upstream ZFS bug #10593, #11682, upstream fix:

commit 296a4a369bc1078a694f88570972330985b3b1b8
Author: Chunwei Chen <email address hidden>
Date: Fri Mar 19 22:53:31 2021 -0700

    Fix zfs_get_data access to files with wrong generation

This patch fixes this issue by record the generation number during
zfs_log_write, so zfs_get_data can check if the object is valid.

The fix is already in Ubuntu ZFS 2.0.6 in Impish. The fix is a relatively simple backport with only minor backporting effort required in moving some of the module specific upstream changes into the pre-renamed module specific os specific parts of the ZFS module.

[Test Plan]

This is difficult to reproduce, see https://github.com/openzfs/zfs/issues/10593

The fix has been tested with the complete ubuntu autotest ZFS regression tests that exercise ZFS core functionality (smoke tests), file system POSIX compliance (fs tests), extra XFS tests and every mount option with stress-ng file I/O stress tests.

[Where problems could occur]

This fix could regress the ZIL (ZFS intent log) commit writes since it modifies this code to now checks for generation numbers. Testing with ZFS autotests with intent logs have not found any regressions, but there is a risk that the ZIL functionality is affected. This could lead to corruption.

------

As mentioned at https://answers.launchpad.net/ubuntu/+source/zfs-linux/+question/696659, the following pull request fixes a panic acquiring a lock: https://github.com/openzfs/zfs/pull/11682

This fix does not appear to be a part of the latest Ubuntu 20.04 release:

zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.12
  Candidate: 0.8.3-1ubuntu12.12
  Version table:
 *** 0.8.3-1ubuntu12.12 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12.9 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     0.8.3-1ubuntu12 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

Of note is this issue where the bug is reported against the Ubuntu 0.8.3 packages: https://github.com/openzfs/zfs/issues/10642

Revision history for this message
Colin Ian King (colin-king) wrote :

Upstream commit:

commit 296a4a369bc1078a694f88570972330985b3b1b8
Author: Chunwei Chen <email address hidden>
Date: Fri Mar 19 22:53:31 2021 -0700

    Fix zfs_get_data access to files with wrong generation

Changed in zfs-linux (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in zfs-linux (Ubuntu Impish):
status: Triaged → Fix Released
Changed in zfs-linux (Ubuntu Focal):
importance: Undecided → High
Changed in zfs-linux (Ubuntu Hirsute):
importance: Undecided → High
status: New → In Progress
Changed in zfs-linux (Ubuntu Focal):
status: New → In Progress
assignee: nobody → Colin Ian King (colin-king)
Changed in zfs-linux (Ubuntu Hirsute):
assignee: nobody → Colin Ian King (colin-king)
Revision history for this message
Colin Ian King (colin-king) wrote :

I've uploaded the packages for SRU, I won't be around to handle the rest of the SRU process though.

description: updated
Changed in zfs-linux (Ubuntu Impish):
assignee: nobody → Colin Ian King (colin-king)
assignee: Colin Ian King (colin-king) → nobody
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in zfs-linux (Ubuntu Hirsute):
assignee: Colin Ian King (colin-king) → Dimitri John Ledkov (xnox)
Changed in zfs-linux (Ubuntu Impish):
status: Fix Released → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Andrew, or anyone else affected,

Accepted zfs-linux into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.0.2-1ubuntu5.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Hirsute):
status: In Progress → Fix Committed
Revision history for this message
Brian Murray (brian-murray) wrote :

Inspecting the source code of zfs-linux for Impish and the patch uploaded for Hirsute it looks to me like this is already fixed in Impish so I'm setting this to Fix Released.

Changed in zfs-linux (Ubuntu Impish):
status: In Progress → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Andrew, or anyone else affected,

Accepted zfs-linux into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.3-1ubuntu12.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in zfs-linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Andrew Berry (andrewberry) wrote :

Thanks for your prompt work on this.

I enabled the proposed repository and have upgraded the zfs packages. As this fix is in the zfs kernel module, I thought I would need to update the kernel packages too:

```
$ dpkg-query -S /lib/modules/5.4.0-89-generic/kernel/zfs/zfs.ko
linux-modules-5.4.0-89-generic: /lib/modules/5.4.0-89-generic/kernel/zfs/zfs.ko
```

I upgraded to `linux-image-5.4.0-90-generic`, but the module version looks to be old:

```
[ 4.366482] ZFS: Loaded module v0.8.3-1ubuntu12.12, ZFS pool version 5000, ZFS filesystem version 5

$ uname -r
5.4.0-90-generic
```

I would have expected that version to be `0.8.3-1ubuntu12.14`. Any suggestions?

```
$ dpkg -l | grep zfs
ii libzfs2linux 0.8.3-1ubuntu12.14 amd64 OpenZFS filesystem library for Linux
ii zfs-auto-snapshot 1.2.4-2 all ZFS automatic snapshot service
ii zfs-initramfs 0.8.3-1ubuntu12.14 amd64 OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.8.3-1ubuntu12.14 amd64 OpenZFS Event Daemon
ii zfsutils-linux 0.8.3-1ubuntu12.14 amd64 command-line tools to manage OpenZFS filesystems
```

```
$ dpkg -l | grep linux- | grep -v '^rc'
ii linux-base 4.5ubuntu3.6 all Linux image base package
ii linux-firmware 1.187.19 all Firmware for Linux kernel drivers
ii linux-image-5.4.0-88-generic 5.4.0-88.99 amd64 Signed kernel image generic
ii linux-image-5.4.0-89-generic 5.4.0-89.100 amd64 Signed kernel image generic
ii linux-image-5.4.0-90-generic 5.4.0-90.101 amd64 Signed kernel image generic
ii linux-image-generic 5.4.0.89.93 amd64 Generic Linux kernel image
ii linux-modules-5.4.0-88-generic 5.4.0-88.99 amd64 Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
ii linux-modules-5.4.0-89-generic 5.4.0-89.100 amd64 Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
ii linux-modules-5.4.0-90-generic 5.4.0-90.101 amd64 Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
ii linux-modules-extra-5.4.0-88-generic 5.4.0-88.99 amd64 Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
ii linux-modules-extra-5.4.0-89-generic 5.4.0-89.100 amd64 Linux kernel extra modules for version 5.4.0 on 64 bit x86 SMP
```

Changed in zfs-linux (Ubuntu Focal):
assignee: Colin Ian King (colin-king) → nobody
assignee: nobody → Dimitri John Ledkov (xnox)
Mathew Hodson (mhodson)
Changed in zfs-linux (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 2.0.2-1ubuntu5.4

---------------
zfs-linux (2.0.2-1ubuntu5.4) hirsute; urgency=medium

  * Fix zfs_get_data access to files causing panics (LP: #1946686)
   - debian/patches/4900-Fix-zfs_get_data-access-to-files-with-wrong-generati.patch
     backport from upstream ZFS commit 296a4a369bc1078a694f88570972330985b3b1b8
     ("Fix zfs_get_data access to files with wrong generation")

 -- Colin Ian King <email address hidden> Tue, 12 Oct 2021 12:34:17 +0100

Changed in zfs-linux (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Andy Whitcroft (apw) wrote : Update Released

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

riscv64 ftbfs is not a regression, as we don't build it in updates either.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

For focal this appears to be required and fixes things for people https://github.com/openzfs/zfs/issues/10642

I will rerun regression suites on focal with latest kernel, and if that passes, will mark this as verification done.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I have completed regression testing with this zfs update in proposed, on focal ga v5.4 kernel, including stress tests and zsys tests. It appears to be stable and working correctly.

We should release this sru, even though I personally cannot reproduce and confirm that the stated issue is fixed. (i.e. https://github.com/openzfs/zfs/issues/10642 / bug description)

$ cat /sys/module/zfs/version
0.8.3-1ubuntu12.14

$ uname -r
5.4.0-110-generic

Note that testing hwe kernels is not interesting as they have zfs module of newer versions built-in.

tags: added: verification-done verification-done-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package zfs-linux - 0.8.3-1ubuntu12.14

---------------
zfs-linux (0.8.3-1ubuntu12.14) focal; urgency=medium

  * Fix zfs_get_data access to files causing panics (LP: #1946686)
   - debian/patches/4920-Fix-zfs_get_data-access-to-files-with-wrong-generati.patch
     backport from upstream ZFS commit 296a4a369bc1078a694f88570972330985b3b1b8
     ("Fix zfs_get_data access to files with wrong generation")

 -- Colin Ian King <email address hidden> Tue, 12 Oct 2021 14:02:12 +0100

Changed in zfs-linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.