zfs-2.1.4+ sru

[Dimitri John Ledkov]

[Impact]

 * Upstream stable point release update with bugfixes, performance fixes, and newer kernel support as needed already in the OEM kernel and will be needed in the future HWE kernels.

[Test Plan]

 * autopkgtest pass

 * kernel regression zfs testsuite pass

 * zsys integration test pass

[Where problems could occur]

 * The stable branches maintain api/abi. Certain bugfixes do change userspace visible behavior of either succeeeding (when previously operations failed), or return errors when previously succeeding in error. For example there are changes when unlinking files in full volumes; changes to fallocate behaviour, etc. Overall they are minor corner cases, and bugfixes to correct the bahaviour to what is universally expected and how things behave on other filesystems (i.e. ext4).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Regression & zsys testing passed out of bileto PPA, submitting for SRU.

[Andy Whitcroft (apw)]

Hello Dimitri, or anyone else affected,

Accepted zfs-linux into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.1.4-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.1.4-0ubuntu0.1

---------------
zfs-linux (2.1.4-0ubuntu0.1) jammy; urgency=medium

  * New upstream point release (LP: #1969482):
    - bufixes
    - performance fixes
    - v5.17 & v5.18 compat
    - scrub services
    - fix fallocate LP: #1969247

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Apr 2022 15:44:23 +0100

[Josef Rypáček (jrypacek)]

I've set up proposed and installed following. It does NOT work as expected.

I used following command to install zfs / kernel packages and dependencies and it resulted in installation / upgrade of packages mentioned below.

apt install zfsutils-linux zfs-initramfs zfs-zed linux-generic linux-image-generic-hwe-22.04 linux-firmware linux-generic-hwe-20.04

Install:
linux-image-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-headers-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-modules-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic),
linux-headers-5.15.0-33:amd64 (5.15.0-33.34, automatic),
linux-modules-extra-5.15.0-33-generic:amd64 (5.15.0-33.34, automatic)

Upgrade:
libnvpair3linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
libuutil3linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
libzpool5linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
libzfs4linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
zfsutils-linux:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
zfs-zed:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1),
zfs-initramfs:amd64 (2.1.2-1ubuntu3, 2.1.4-0ubuntu0.1)
linux-headers-generic:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-generic:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-image-generic:amd64 (5.15.0.30.33, 5.15.0.33.36)
linux-image-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-headers-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-generic-hwe-22.04:amd64 (5.15.0.30.33, 5.15.0.33.36)
linux-generic-hwe-20.04:amd64 (5.15.0.30.33, 5.15.0.33.36),
linux-firmware:amd64 (20220329.git681281e4-0ubuntu3, 20220329.git681281e4-0ubuntu3.2)

Then I got following system after reboot

root@XXX:~# uname -a
Linux nas139 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@XXX:~# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

I still have old zfs-kmod a still very low ZFS performance on encrypted filesystem.
Did I do something wrong or there is missing updated zfs-kmod?

[Dimitri John Ledkov (xnox)]

Hi, the verification you have performed is incorrect. You installed kernel and using zfs.ko as built into the kernel, which vendors zfs.ko from release/updates pockets only.

To test, zfs.ko from proposed, one has to install the dkms package, build it, enroll MOK (if secureboot is on, and MOK key is not already enrolled), and reboot.

[Josef Rypáček (jrypacek)]

I've installed zfs-dkms and it did the work. I can confirm system works as expected and performance is higher now.

# uname -a
Linux nas139 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1

[Brian Murray (brian-murray)]

I'm setting this back to verification-needed as we still need to hear back from the kernel team about this part of the test plan:

 * kernel regression zfs testsuite pass

 * zsys integration test pass

[Harshal Prakash Patankar (pharshalp)]

I tested this version and it does fix a (severe) performance regression as mentioned here (#1977699) https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1977699. Would this be released before 21.10 reaches EOL?

[David Griffin (habilain)]

I can also confirm that the fixes in -proposed fixes some severe performance issues on my system, with an encrypted ZFS root filesystem.

It should be noted that if you're using ZFS on an Intel machine with < 4 physical cores, these performance regressions are severe enough to cause the entire machine to noticeably stutter and freeze when the disk is accessed - so this is a fairly serious performance regression for a lot of laptop users!

I think the reason why it's so severe on this configuration is due to ZFS read/write spawning 4 tasks, which have a niceness of -20 causing the userspace to be starved (and perhaps security mitigations stopping/hindering the userspace from using hyperthreading).

[Dimitri John Ledkov (xnox)]

kernel regression zfs testsuite has passed with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms modules installed with stock zfs modules removed from the system. smoke, xfs-generic, fstest, stress all completed without any errors.

remaining testing is zsys installations and snapshots.

[Dimitri John Ledkov (xnox)]

unencrypted zsys install worked, snapshotted, and booted into snapshot with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms module.

[Dimitri John Ledkov (xnox)]

encrypted zsys install worked, snapshotted, and booted into snapshot with v5.15.0-40-generic kernel and 2.1.4-0ubuntu0.1 dkms module.

[Dan Podeanu (pdan)]

Is there an ETA for releasing the fix to Jammy?

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.1.4-0ubuntu0.1

---------------
zfs-linux (2.1.4-0ubuntu0.1) jammy; urgency=medium

  * New upstream point release (LP: #1969482):
    - bufixes
    - performance fixes
    - v5.17 & v5.18 compat
    - scrub services
    - fix fallocate LP: #1969247

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Apr 2022 15:44:23 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dan Podeanu (pdan)]

I need an advice please:

Does this mean we need to switch to zfs-dkms instead of stock zfs to get the fix? Is this the long-term recommended solution?

As it stands today, with stock zfs, the zfs-kmod version is 2.1.2 hence contains the performance regression. Example on an up-to-date Jammy:

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

[Dimitri John Ledkov (xnox)]

@pdan

Ubuntu kernels vendor and build zfs.ko from the updates pocket. Meaning the next sru cycle of Ubuntu kernels will have the updated zfs.

If you want to receive zfs.ko ahead of time, you can install zfs-dkms for a little bit, until it lands in the kernel builds. Or just wait a little bit longer.

[Dan Podeanu (pdan)]

@dmitri

Thank you! Quick question, the Jammy kernel version was just bumped, but it still does not include 2.1.4 zfs module

# uname -a
Linux arpa 5.15.0-41-generic #44-Ubuntu SMP Wed Jun 22 14:20:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.2-1ubuntu3

[Dimitri John Ledkov (xnox)]

yes..... because that's the old sru cycle. the kernel was already built on 22nd of June and was undergoing validation. This zfs update was released to updates after that, on 28th of June.

the next sru cycle that will pick up updated zfs has not started yet.

[Dan Podeanu (pdan)]

Got it, thank you. It sounds like testing encrypted zfs performance could perhaps be added to the validation suite for kernels.

[Dan Podeanu (pdan)]

The updated kernel was released, and it fixes the performance regression. Thank you!

# uname -a
Linux arpa 5.15.0-43-generic #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

# zfs --version
zfs-2.1.4-0ubuntu0.1
zfs-kmod-2.1.4-0ubuntu0.1

# grep . /sys/module/icp/parameters/*impl*
/sys/module/icp/parameters/icp_aes_impl:cycle [fastest] generic x86_64 aesni
/sys/module/icp/parameters/icp_gcm_impl:cycle [fastest] avx generic pclmulqdq

# dd if=14GBfile.tmp of=/dev/null bs=1M
13411+1 records in
13411+1 records out
14062902185 bytes (14 GB, 13 GiB) copied, 12.6139 s, 1.1 GB/s