crash in openzfs - 2.2.2 not supported on 6.8

[Erik Hortsch]

SRU Justification

[Impact]

* Seeing crash due to use-after-free. Logs included below SRU justification

[Fix]

* Backport 38c0324c0fb6: "Linux: Fix zfs_prune panics"

[Test Plan]

* Compile tested
* Run through autopkgtest regression tests

[Regression potential]

* Changes isolated, minimal regression risk. Changes already in upstream ZFS

Crash report on openzfs: https://github.com/openzfs/zfs/issues/16482

> You should probably report this to Ubuntu or try it on 2.2.5 vanilla, 2.2.2 doesn't claim to work on 6.8

Distribution Version 24.04
Kernel Version 6.8.0-41-generic
Architecture intel x86-64
OpenZFS Version zfs-2.2.2-0ubuntu9, zfs-kmod-2.2.2-0ubuntu9

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: zfsutils-linux 2.2.2-0ubuntu9
ProcVersionSignature: Ubuntu 6.8.0-41.41-generic 6.8.12
Uname: Linux 6.8.0-41-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.28.1-0ubuntu3.1
Architecture: amd64
CasperMD5CheckResult: pass
Date: Mon Aug 26 21:30:11 2024
InstallationDate: Installed on 2024-08-16 (11 days ago)
InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Release amd64 (20240424)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: zfs-linux
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.sudoers.d.zfs: [inaccessible: [Errno 13] Permission denied: '/etc/sudoers.d/zfs']

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Aleksandr Mikhalitsyn (mihalicyn)]

I have some evidences that potentially we have even more serious trouble with ZFS on Ubuntu Noble 6.8 kernel.

Some time ago LXD team has got a report about LXCFS processes crashes:
https://github.com/lxc/lxcfs/issues/644

After spending really a lot of time checking everything from our side (in LXCFS) I started to have a feeling that problem can be with the kernel (memory corruption).

Issue reported was using all standard Ubuntu Noble packages - ZFS from Noble and kernel from Noble. Also LXD v5.21.1 LTS snap.

I've instructed him to try an upstream version of ZFS module with the same kernel (no luck there).
Then, I've instructed user to try an recent upstream Linux kernel version (same, no luck).

Then user downgraded to 6.6.x series kernel + upstream recent ZFS. And LXCFS stopped crashing and everything works just fine for him. And that's without any changes on the LXD snap side (LXCFS, LXC, LXD - all the same!)

Now we have another report:
https://github.com/canonical/lxd/issues/14178

I decided to share this information in this issue instead of filling a new one so we can collect all the problems with ZFS on a new kernels in one place.

To conclude: all points that we have a bug in ZFS which leads to a kernel/userspace memory corruption...

[John Cabaj (john-cabaj)]

Acknowledging that I'm looking into this alongside some ZFS bugs for other releases. Any concrete steps to reproduce (preferably via LXD) are welcome, though I'll search through the various issue links raised above to see if there's any reproducer there as well.

[Aleksandr Mikhalitsyn (mihalicyn)]

Hey John!

Of course, if I have any piece of information about this one (like reproducer) I'll share it with you. But for now, we only have these two LXD issues (links are in comment #3) and that's it.

Also you may look into this discussion:
https://github.com/openzfs/zfs/issues/16324

Also, I'm curious if we have any plans to update our Noble's ZFS version to 2.2.6, as it contains a lot of fixes like:
https://github.com/openzfs/zfs/commit/fa2480f5b3bc1d265304a950c1b5ab05497853cd
https://github.com/openzfs/zfs/commit/859f906a4b4dfc132115f10f5d372ef4281a6479
https://github.com/openzfs/zfs/commit/0f9457d1dd7a56ceccda962433281ef946132b5f
and many others.

Kind regards,
Alex

[John Cabaj (john-cabaj)]

Seems there a couple of potential fixes upstream - https://github.com/openzfs/zfs/pull/16770/commits/96c82faf731594f4fadf1166c37749e8e05cd9b2 and https://github.com/openzfs/zfs/pull/16788/commits/1c6da1a60a37730fa635d02f45d266e7177cc6c6.

Will try to get a package together tonight and run through autopkgtests. But I don't have a reproducer at the moment.

[Andre Toerien (athepeanut4)]

I have been experiencing random process crashes and kernel panics ever since I upgraded to 24.04. The first was due to a btrfs bug (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2080039), but they've continued even after the fix for that was released, although much less frequently.

The crashes seem to generally occur when the system is under I/O pressure, on both ZFS and non-ZFS filesystems. Used memory is <6GB/16GB, but the remainder is basically all used for disk caching - so it's effectively under memory pressure as well.

I set up kdump after the first crash I got since the above bug was fixed, and figured out how to do basic dump analysis. From what I can tell, the kernel panics are either page fault related, or illegal operation exceptions (i.e. memory corruption).

The process crashes are always for the process causing the high disk usage (a python downloader script), and are either SIGILL's or SIGSEGV's.

From a comment above I've just found https://github.com/canonical/lxd/issues/14178#issuecomment-2402322735, and have now enabled both KFENCE and SLUB debugging, so we'll see if I get any dmesg error messages.

So that is all to say, an upgrade to 2.2.6 on noble would be very much appreciated. I'd also be happy to test anything and try to force a crash.

Thanks!

[John Cabaj (john-cabaj)]

Including 2.2.2-0ubuntu10.debdiff for updates.

[John Cabaj (john-cabaj)]

Including 2.2.6-1ubuntu1.1.debdiff for updates.

[Heitor Alves de Siqueira (halves)]

Fixed in 2.2.7 release, which is now in Plucky

[Timo Aaltonen (tjaalton)]

Hello Erik, or anyone else affected,

Accepted zfs-linux into oracular-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.2.6-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-oracular to verification-done-oracular. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-oracular. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Erik, or anyone else affected,

Accepted zfs-linux into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.2.2-0ubuntu9.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted zfs-linux (2.2.6-1ubuntu1.1) for oracular have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/2.2.6-1ubuntu1.1 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/oracular/update_excuses.html#zfs-linux

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[John Cabaj (john-cabaj)]

No cases of arc_prune panics when running ZFS testsuite. Additionally, the sole available reproducer upstream indicated that the patch fixed the issue - https://github.com/openzfs/zfs/issues/16324#issuecomment-2492964532.

[Andreas Hasenack (ahasenack)]

Hello @john-cabaj,

thanks for trying out the proposed packages. Unfortunately your verification in comment #14 is missing crucial information. As requested in comment #12:

  If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the
  package you tested, what testing has been performed on the package and change the tag from verification-
  needed-noble to verification-done-noble.

Same for the oracular verification.

[John Cabaj (john-cabaj)]

Dropping a new comment below so the comment history makes sense:

No cases of arc_prune panics when running ZFS testsuite against version 2.2.6-1ubuntu1.1 for Oracular or 2.2.2-0ubuntu9.2 for Noble. Additionally, the sole available reproducer upstream indicated that the patch fixed the issue - https://github.com/openzfs/zfs/issues/16324#issuecomment-2492964532.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.2.6-1ubuntu1.1

---------------
zfs-linux (2.2.6-1ubuntu1.1) oracular; urgency=medium

  * debian/patches/0001-Linux-Fix-zfs_prune-panics.patch
    - crash in openzfs - 2.2.2 not supported on 6.8 (LP: #2077926)
  * debian/patches/0002-zfs_log-add-flex-array-fields-to-log-record-structs.patch
  * debian/patches/0003-lua-add-flex-array-field-to-TString-type.patch
    - memcpy: detected field-spanning write (size 8) of
      single field "lr + 1" (LP: #2082060)

 -- John Cabaj <email address hidden> Mon, 09 Dec 2024 18:00:06 -0600

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.2.2-0ubuntu9.2

---------------
zfs-linux (2.2.2-0ubuntu9.2) noble; urgency=medium

  [ John Cabaj ]
  * debian/patches/0001-Improve-performance-for-zpool-trim-on-linux.patch
  * debian/patches/0002-vdev_disk-ensure-trim-errors-are-returned-immediatel.patch
    - task txg_sync:696 blocked (LP: #2081678)
  * debian/patches/0003-Autotrim-High-Load-Average-Fix.patch
    - Activating autotrim results in high load average due to
      uninterruptible threads (LP: #2057693)
  * debian/patches/0004-Linux-Fix-zfs_prune-panics.patch
    - crash in openzfs - 2.2.2 not supported on 6.8 (LP: #2077926)

  [ Heitor Alves de Siqueira ]
  * Fix "field-spanning write" errors on zpool import (LP: #2082060):
    - d/p/4701-zfs_log-add-flex-array-fields-to-log-record-structs.patch
    - d/p/4702-lua-add-flex-array-field-to-TString-type.patch

 -- John Cabaj <email address hidden> Mon, 09 Dec 2024 14:00:32 -0600

[Aleksandr Mikhalitsyn (mihalicyn)]

Dear friends,

Will this land to ZFS from Jammy? Jammy users can also update to 6.8 kernel, but they have ZFS 2.1.5 and have no chance to get benefit from these fixes.

It is obvious that ZFS 2.1.5 is even worse with 6.8.x kernels than ZFS 2.2.2.