upgrade zfs-linux to 2.2.0 final

[Dimitri John Ledkov]

[ Impact ]

 * Potential data loss with zfs 2.2.0-rc3 resolved in -rc5, proposal to update to final

 * Ubuntu Mantic shipped 2.2.0-rc3 with approximately 10 additional cherrypicks, or kernel team created fixes that got contributed & accepted upstream. At the time this was the only way to get zfs working with v6.5 kernels and also drop the need for shiftfs (due to zfs impovements). Since us shipping this package, upstream has identified and fixed multiple small bugfixes in the subsequent RC and the final releases, including one bug fix that can lead to potential data loss.

 * The 2.2.0 release branch was frozen for a long time already, and outstanding number of commits of fixes that Mantic does not have is less than 30 small patches.

 * Proposal to upgrade our build to 2.2.0 final, pick up all the regression fixes, and drop all the cherrypicked patches that enable v6.5 support. This will give us the best kernel driver to support in the runnup to next Ubuntu LTS.

[ Test Plan ]

 * autopkgtest pass

 * kernel regression zfs testsuite pass

 * zsys integration test pass

 * LXD support retested

[ Where problems could occur ]

 * LXD snap in edge shipped zfs tooling of RC5 version until 16th October when they upgraded to 2.2.0 final, there are no kernel-userspace incompatiblities between RC & final, but we should explicitly test this.

[ Other Info ]

 * Upstream is alerting us to the potential data loss and requesting upgrade to 2.2.0-rc5 or better.

[ Abbriviated changes being introduced ]

$ git log --oneline 4a104ac047..95785196f2 -- cmd/ lib/ module/os/linux/ | grep -v compat
810fc49a3e Ensure we call fput when cloning fails due to different devices.
a80e1f1c90 zvol: Temporally disable blk-mq
33d7c2d165 import: require force when cachefile hostid doesn't match on-disk
8015e2ea66 Add '-u' - nomount flag for zfs set
c53bc3837c Improve the handling of sharesmb,sharenfs properties
e9dc31c74e Update the behavior of mountpoint property
608741d062 Report ashift of L2ARC devices in zdb
0ce1b2ca19 Invoke zdb by guid to avoid import errors
0aabd6b482 ZIL: Avoid dbuf_read() in ztest_get_data()
a199cac6cd status: report pool suspension state under failmode=continue
729507d309 Fix occasional rsend test crashes
3af63683fe cmd: add 'help' subcommand to zpool and zfs
9aa1a2878e Fix incorrect expected error in ztest
f7a07d76ee Retire z_nr_znodes
54c6fbd378 zed: Allow autoreplace and fault LEDs for removed vdevs
32949f2560 Relax error reporting in zpool import and zpool split
63159e5bda checkstyle: fix action failures
e99e684b33 zed: update zed.d/statechange-slot_off.sh
d19304ffee zed: Add zedlet to power off slot when drive is faulted
92f095a903 copy_file_range: fix fallback when source create on same txg
895cb689d3 zfs_clone_range should return a descriptive error codes
6bdc7259d1 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1
df8c9f351d ZIL: Second attempt to reduce scope of zl_issuer_lock.
0ae7bfc0a4 zpool_vdev_remove() should handle EALREADY error return
bd1eab16eb linux: zfs: ctldir: set [amc]time to snapshot's creation property
c47f0f4417 linux/copy_file_range: properly request a fallback copy on Linux <5.3
12f2b1f65e zdb: include cloned blocks in block statistics

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.2.0-0ubuntu1

---------------
zfs-linux (2.2.0-0ubuntu1) noble; urgency=medium

  * New upstream release. LP: #2040181
    - drop patches applied upstream
    - fix potential data loss
    - upgrade to 2.2.0 final
    - adjust symbols

 -- Dimitri John Ledkov <email address hidden> Tue, 24 Oct 2023 13:25:21 +0100

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted zfs-linux into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.2.0-0ubuntu1~23.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Birgit Edel (biredel)]

Extra step needed when testing these proposed packages against kernels providing the rc:
dkms and dpkg disagree on version sort order.

# dpkg prioritizes tilde
dpkg --compare-versions '2.2.0-0ubuntu1~23.10' gt '2.2.0~rc3-0ubuntu4' ; echo $?

# dkms determines dash comes first
dpkg-reconfigure zfs-dkms
[..]
zfs.ko.zst:
Running module version sanity check.
Error! Module version 2.2.0-0ubuntu1~23.10 for zfs.ko.zst
is not newer than what is already found in kernel 6.5.0-10-generic (2.2.0~rc3-0ubuntu4).
You may override by specifying --force.

Doing as suggested works:
sudo dkms install --force zfs/2.2.0

Suppose this cannot be changed (breaking compat) in dkms, but looking at the dkms code, assigning monotonic (under both regimen) versions for the next pre-release versions seems possible.

[Dimitri John Ledkov (xnox)]

dkms is supposed to use dpkg version comparison on ubuntu though =/

[Dimitri John Ledkov (xnox)]

(forked dkms buggy version comparison to https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/2044137 )

[Dimitri John Ledkov (xnox)]

Otherwise, this bug report is verified and ready for releases. (and all kernels will rebuild with new driver anyway, thus version comparison will soon become 'identical already installed' and thus all rc versions will be flushed from the archive and the kernels)

[Mauricio Faria de Oliveira (mfo)]

Hi Dimitri,

Could you please provide more evidence/details of the test plan execution, considering this SRU is related to a filesystem?

```
[ Test Plan ]

 * autopkgtest pass

 * kernel regression zfs testsuite pass

 * zsys integration test pass

 * LXD support retested
```

The autopkgtests are OK [1], but the remaining items aren't documented.

[1] https://ubuntu-archive-team.ubuntu.com/proposed-migration/mantic/update_excuses.html#zfs-linux

[Dimitri John Ledkov (xnox)]

Note the 10.30 cycle mantic kernels were built with this zfs-linux dkms prebuilt, vendored in, and passing kernel regresstion testing, adt-matrix testing and so on, for all kernels.

Mantic kernels that contain this update of zfs.ko have now been mostly released.

 * autopkgtest pass

As seen on https://ubuntu-archive-team.ubuntu.com/proposed-migration/mantic/update_excuses.html#zfs-linux
And due to lack of automated comments of autopkgtest regressions
And also via adt matrix at for example https://kernel.ubuntu.com/adt-matrix/mantic-linux-meta.html for zfs-linux results with generic kernel, but also all other mantic kernels.

 * kernel regression zfs testsuite pass

(Canonical only URL i believe) http://10.246.75.167/2023.10.30/tc-stats-lvl1.html shows all RT testing stats for the 2013.10.30 cycle, and you can see that zfs ones are all green (apart from one stress-ng test case which seems to have passed, but failed to clean up with a timeout, and is being investigated). Clicking through to the zfs test case name you can see break-down of tests per release / kernel flavour / target hardware / arch etc. For this SRU only mantic one matters.

 * zsys integration test pass

Tested that existing installs upgrade fine, and new ones are also possible by upgrading zfs-linux userspace tooling in live desktop session to proposed.

 * LXD support retested

That was done by lxd team, and they have shipped upgrade to 2.0 final inside lxd snap already.

In practice this SRU is already released to most users via LXD and via kernel upgrades. The change this sru release will introduce is userspace tooling update, which was minor in this update.

Note there will be a new zfs-linux sru, right after this one to address the recently identified and patched up issues. See https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2044657

It is still worth it to release this sru, before staging the next one in proposed.

[Mauricio Faria de Oliveira (mfo)]

Hi Dimitri,

Thanks for the details of the test plan execution!

That covers the kernel module/filesystem code quite well, thanks,
and there's the userspace tooling changes left, as you mentioned.

> In practice this SRU is already released to most users via LXD and via kernel upgrades.
> The change this sru release will introduce is userspace tooling update, which was minor in this update.

I looked up the changes to userspace commands (in subdir `zfs/cmd/`),
and would agree they are (mostly) minor, with a few notes to have
documented (and for others to review), but IMHO, none are blocking
for the release of this SRU.

cheers,
Mauricio

Related commits:

 $ git log --oneline zfs-2.2.0-rc3..zfs-2.2.0 -- cmd/
 33d7c2d165c2 import: require force when cachefile hostid doesn't match on-disk
 8015e2ea66b4 Add '-u' - nomount flag for zfs set
 c53bc3837cb6 Improve the handling of sharesmb,sharenfs properties
 e9dc31c74e7b Update the behavior of mountpoint property
 608741d062fe Report ashift of L2ARC devices in zdb
 0ce1b2ca1930 Invoke zdb by guid to avoid import errors
 0aabd6b48228 ZIL: Avoid dbuf_read() in ztest_get_data()
 3af63683fe07 cmd: add 'help' subcommand to zpool and zfs
 9aa1a2878ea9 Fix incorrect expected error in ztest
 54c6fbd378ea zed: Allow autoreplace and fault LEDs for removed vdevs
 32949f2560bf Relax error reporting in zpool import and zpool split
 63159e5bda1c checkstyle: fix action failures
 e99e684b337b zed: update zed.d/statechange-slot_off.sh
 d19304ffeec5 zed: Add zedlet to power off slot when drive is faulted
 df8c9f351dab ZIL: Second attempt to reduce scope of zl_issuer_lock.
 12f2b1f65e91 zdb: include cloned blocks in block statistics

Spotted changes/commits:

1) Config option with typo

commit d19304ffeec50ebc02cf4496c14e8945c74fb76a
Author: Tony Hutter <email address hidden>
Date: Thu Aug 24 11:59:03 2023 -0700

    zed: Add zedlet to power off slot when drive is faulted

    If ZED_POWER_OFF_ENCLOUSRE_SLOT_ON_FAULT is enabled in zed.rc, then
    power off the drive's slot in the enclosure if it becomes FAULTED.
    This can help silence misbehaving drives. This assumes your drive
    enclosure fully supports slot power control via sysfs.

This new option is disabled by default, good.
However, the option has a typo, so I submitted patch [1] upstream,
and it would be nice if we pick it up later on.

[1] https://github.com/openzfs/zfs/pull/15651

...

2) zpool import/split no longer reflect errors to mount/share

commit 32949f2560bf35ec86dfa5d984514908e0eb3ecc
Author: Umer Saleem <email address hidden>
Date: Sat Sep 2 05:25:11 2023 +0500

    Relax error reporting in zpool import and zpool split

    For zpool import and zpool split, zpool_enable_datasets is called
    to mount and share all datasets in a pool. If there is an error
    while mounting or sharing any dataset in the pool, the status of
    import or split is reported as failure. However, the changes do
    show up in zpool list.

    This commit updates the error reporting in zpool import and zpool
    split path. More descriptive messages are shown to user in case
    there is an error during mount or share. Errors in mount or share
    do not effect...

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.2.0-0ubuntu1~23.10

---------------
zfs-linux (2.2.0-0ubuntu1~23.10) mantic; urgency=medium

  * New upstream release. LP: #2040181
    - drop patches applied upstream
    - fix potential data loss
    - upgrade to 2.2.0 final
    - adjust symbols

 -- Dimitri John Ledkov <email address hidden> Tue, 24 Oct 2023 13:25:21 +0100