Broken atime in 2.2.2-0ubuntu9 (noble's version)

[Jessica Clarke]

SRU Justification

[Impact]

* 2.2.2-0ubuntu9 has the 6.7 compat series, including https://github.com/openzfs/zfs/commit/db4fc559cc1931b9219e62aa505cd5c51a17b232, but does not include the fix for that commit (affects both pre-6.7 and post-6.7),
https://github.com/openzfs/zfs/commit/f0bf7a247dbb030d68c7fd2b5526dd111cc775d0. This manifests for me on the current (6.8.0-40.40~22.04.3) 22.04 HWE kernel as writes to a file resetting atime back to 0 (reads still seem to work as expected though given that state), which of course breaks real-world things (in my case, I can no longer successfully run LLVM's test suite).

Thanks to CONFIG_INIT_STACK_ALL_ZERO=y being the default it seems like this isn't a kernel memory disclosure and is instead a deterministic use of 0, so I don't believe this to be a blatant security vulnerability. This also explains the behaviour observed above.

This needs fixing in noble, and then propagating back through whatever kernels are pulling in this version, since linux-modules vendors copies of the pre-built DKMS modules, with the current 6.8 HWE kernel using 2.2.2-0ubuntu9 explicitly.

[Fix]

* Include f0bf7a247dbb: "[PATCH] Linux 6.7 compat: zfs_setattr fix atime update" from upstream ZFS as quilt patch.

[Test Case]

* Build tested against Noble generic
* Ran autopkgktests for regressions, none encountered
* Ran the following test case:

Using zfs-linux 2.2.2-0ubuntu9
$ cd /path/to/zfs
$ rm -f foo
$ touch foo
$ stat foo
$ # observe initial access time of 0 since epoch (1970-01-01)

Using zfs-linux 2.2.2-0ubuntu9.1
$ cd /path/to/zfs
$ rm -f foo
$ touch foo
$ stat foo
$ # observe initial access time same as modify/change/birth

Also see results attached as "zfs-bad" and "zfs-good", respectively

[Where things could go wrong]

* Low chance of regression, isolated instantiation and logic fix.
* Addition of time delta check could cause issues during unexpected delay > 2 seconds.

[Jessica Clarke (jrtc27)]

NB: Given the 22.04 HWE kernel is pulling from 24.04, this also affects 24.04 out of the box, as expected, no HWE kernel required to hit it.

[John Cabaj (john-cabaj)]

Updating LP bug to SRU justification format.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

The current test plan doesn't seem to cover what is described in the impact section.

Please update the test plan to include a test for the actual impact this is having, which, from the description, is:

  This manifests for me on the current (6.8.0-40.40~22.04.3) 22.04 HWE kernel as writes to a file resetting
  atime back to 0 (reads still seem to work as expected though given that state),

It should be possible to come up with a test to reproduce the above behavior, and confirm the fix.

[Jessica Clarke (jrtc27)]

My testing (for an atime+relatime filesystem) was:

```
$ cd /path/to/zfs
$ # Test create
$ rm -f foo
$ touch foo
$ stat foo
  File: foo
  Size: 0 Blocks: 2 IO Block: 131072 regular empty file
Device: 27h/39d Inode: 81480314 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 3815/ jrtc4) Gid: ( 3815/ jrtc4)
Access: 1970-01-01 01:00:00.000000000 +0100 # Should be the same as the others
Modify: 2024-09-25 15:05:04.272141094 +0100
Change: 2024-09-25 15:05:04.272141094 +0100
 Birth: 2024-09-25 15:05:04.272141094 +0100
$ # Test access
$ cat foo
$ stat foo
  File: foo
  Size: 0 Blocks: 2 IO Block: 131072 regular empty file
Device: 27h/39d Inode: 81480314 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 3815/ jrtc4) Gid: ( 3815/ jrtc4)
Access: 2024-09-25 15:05:18.971201360 +0100 # This is correct (matches ext4 behaviour)
Modify: 2024-09-25 15:05:04.272141094 +0100
Change: 2024-09-25 15:05:04.272141094 +0100
 Birth: 2024-09-25 15:05:04.272141094 +0100
$ # Test modify
$ touch foo
$ stat foo
  File: foo
  Size: 0 Blocks: 2 IO Block: 131072 regular empty file
Device: 27h/39d Inode: 81480314 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 3815/ jrtc4) Gid: ( 3815/ jrtc4)
Access: 1970-01-01 01:00:00.000000000 +0100 # Should be the same as modify / change
Modify: 2024-09-25 15:05:30.388248172 +0100
Change: 2024-09-25 15:05:30.388248172 +0100
 Birth: 2024-09-25 15:05:04.272141094 +0100
```

That should suffice for reproduction and validation of the fix.

[John Cabaj (john-cabaj)]

Trace showing reproducibility using zfs-linux 2.2.2-0ubuntu9

[John Cabaj (john-cabaj)]

Trace showing non-reproducibility using zfs-linux 2.2.2-0ubuntu9.1

[Andreas Hasenack (ahasenack)]

Hello Jessica, or anyone else affected,

Accepted zfs-linux into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/2.2.2-0ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ying (yingternet)]

It did not fix for me. please check below:

root@zfs-test:/zfs# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
root@zfs-test:/zfs# dpkg -s zfsutils-linux | grep '^Version:'
Version: 2.2.2-0ubuntu9.1
root@zfs-test:/zfs# touch file
root@zfs-test:/zfs# stat file
  File: file
  Size: 0 Blocks: 1 IO Block: 131072 regular empty file
Device: 0,33 Inode: 128 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 1970-01-01 00:00:00.000000000 +0000
Modify: 2024-10-05 08:10:33.558526613 +0000
Change: 2024-10-05 08:10:33.558526613 +0000
 Birth: 2024-10-05 08:10:33.558526613 +0000
root@zfs-test:/zfs# cat file
root@zfs-test:/zfs# stat file
  File: file
  Size: 0 Blocks: 1 IO Block: 131072 regular empty file
Device: 0,33 Inode: 128 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2024-10-05 08:10:39.050528093 +0000
Modify: 2024-10-05 08:10:33.558526613 +0000
Change: 2024-10-05 08:10:33.558526613 +0000
 Birth: 2024-10-05 08:10:33.558526613 +0000
root@zfs-test:/zfs# uname -a
Linux zfs-test 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

[Jessica Clarke (jrtc27)]

Where is your module coming from though? Normally it’s pre-built in linux-modules or an HWE version thereof, so you need to either use the zfs-linux one or get an updated linux-modules.

[Jessica Clarke (jrtc27)]

I can confirm that zfs-dkms 2.2.2-0ubuntu9.1 in noble-proposed fixes this bug, and verified that it was reproducible with zfs-dkms 2.2.2-0ubuntu9, not just the linux-modules binary.

[Ying (yingternet)]

I updated the file: /etc/apt/sources.list.d/ubuntu.sources
to include noble-proposed

Types: deb
URIs: http://archive.ubuntu.com/ubuntu
Suites: noble noble-updates noble-backports noble-proposed
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

then

apt update && apt-get install zfsutils-linux/noble-proposed

Am I missing anything?

[Jessica Clarke (jrtc27)]

That’s the userspace utilities, not the DKMS package.

[Ying (yingternet)]

ok, now I also upgrade the dkms package

apt install zfs-dkms/noble-proposed

after reboot, it works. thanks.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for zfs-linux has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Jessica Clarke (jrtc27)]

> Changed in linux-hwe-6.8 (Ubuntu):
> status: New → Opinion
> status: Opinion → Invalid
> Changed in linux-hwe-6.8 (Ubuntu Noble):
> status: New → Invalid

What's the justification behind this? That is where Ubuntu ships a pre-built copy of zfs-dkms and is how the system I encountered this bug on gets its ZFS kernel module. There does need to be an SRU of any such packages with this SRU'ed zfs-linux otherwise anyone using the pre-built module will still have a broken system.

[Launchpad Janitor (janitor)]

This bug was fixed in the package zfs-linux - 2.2.2-0ubuntu9.1

---------------
zfs-linux (2.2.2-0ubuntu9.1) noble; urgency=medium

  * debian/patches/0008-Linux-6.7-compat-zfs_setattr-fix-atime-update.patch
    - Linux 6.7 support (LP: #2077487)

 -- John Cabaj <email address hidden> Thu, 05 Sep 2024 14:38:42 -0500

[Jessica Clarke (jrtc27)]

(https://git.launchpad.net/ubuntu/+source/linux-hwe-6.8/tree/debian.hwe-6.8/dkms-versions?h=applied/ubuntu/jammy-devel still references 2.2.2-0ubuntu9, for example)

[John Cabaj (john-cabaj)]

Hi Jessica,

You're correct - only the DKMS package itself has the fix to this point, for those installing via DKMS. Kernels that release with the ZFS module bundled need to follow. The Noble and Jammy-HWE 6.8 kernels will contain the updated ZFS module in the upcoming s2024.09.30 kernel cycle, releasing the week of 11-Nov.

John Cabaj

[Chris (saturos+ubu)]

Hi, can someone please clarify when this fix is planned to go out in a Noble kernel?

I just received 6.8.0-49-generic and rebooted and the problem does _not_ seem to be fixed, and I'm really struggling to confirm the workflow & intended patches on Launchpad. But it seems that my ZFS kernel module is still 2.2.2-0ubuntu9, not 9.1.

zfs-2.2.2-0ubuntu9.1
zfs-kmod-2.2.2-0ubuntu9

6.8.0-49-generic #49-Ubuntu

[John Cabaj (john-cabaj)]

This should be fixed in 6.8.0-50.51-generic.