arc_summary doesn't work with HWE kernel 5.15

[Simon Déziel]

[ Impact ]
The arc_summary command doesn't work on HWE kernels for Focal

[ Test Plan ]
1) setup 20.04 with HWE kernel 5.15
2) install `zfsutils-linux`
3) run `arc_summary`, this should not cause any errors

$ arc_summary
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 875, in <module>
    main()
  File "/usr/sbin/arc_summary", line 826, in main
    kstats = get_kstats()
  File "/usr/sbin/arc_summary", line 259, in get_kstats
    with open(PROC_PATH+section, 'r') as proc_location:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/spl/kstat/zfs/xuio_stats'

[ Where problems could occur ]
Given that upstream does not have to deal with two kernel versions like Ubuntu (GA vs HWE), our SAUCE patch will skip any non-existent files under /proc/spl/kstat/zfs. This ensures that arc_summary works as expected under both GA and HWE kernels.

Users expecting the HWE kernel to contain output from xuio_stats like we had in GA will not see it in the ARC summary output. Any new stats that get introduced into the HWE kernel will also be skipped due to this patch, if they're missing from the GA kernel.

--
# Issue description

`arc_summary` no longer works with kernel 5.15. It used to work with previous kernel like 5.13.

# Steps to reproduce

1) setup 20.04 with HWE kernel 5.15
2) install `zfsutils-linux`
3) run `arc_summary`
$ arc_summary
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 875, in <module>
    main()
  File "/usr/sbin/arc_summary", line 826, in main
    kstats = get_kstats()
  File "/usr/sbin/arc_summary", line 259, in get_kstats
    with open(PROC_PATH+section, 'r') as proc_location:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/spl/kstat/zfs/xuio_stats'

Indeed, the xuio_stats file isn't there anymore:
$ ll /proc/spl/kstat/zfs/
total 0
dr-xr-xr-x 21 root root 0 Jul 6 10:49 ./
dr-xr-xr-x 4 root root 0 Jul 6 10:49 ../
-rw-r--r-- 1 root root 0 Jul 6 10:49 abdstats
-rw-r--r-- 1 root root 0 Jul 6 10:49 arcstats
dr-xr-xr-x 20 root root 0 Jul 6 10:49 data/
-rw------- 1 root root 0 Jul 6 10:49 dbgmsg
-rw------- 1 root root 0 Jul 6 10:49 dbufs
-rw-r--r-- 1 root root 0 Jul 6 10:49 dbufstats
dr-xr-xr-x 70 root root 0 Jul 6 10:49 default/
-rw-r--r-- 1 root root 0 Jul 6 10:49 dmu_tx
-rw-r--r-- 1 root root 0 Jul 6 10:49 dnodestats
-rw-r--r-- 1 root root 0 Jul 6 10:49 fletcher_4_bench
-rw-r--r-- 1 root root 0 Jul 6 10:49 fm
-rw-r--r-- 1 root root 0 Jul 6 10:49 import_progress
-rw-r--r-- 1 root root 0 Jul 6 10:49 metaslab_stats
-rw-r--r-- 1 root root 0 Jul 6 10:49 vdev_cache_stats
-rw-r--r-- 1 root root 0 Jul 6 10:49 vdev_mirror_stats
-rw-r--r-- 1 root root 0 Jul 6 10:49 vdev_raidz_bench
-rw-r--r-- 1 root root 0 Jul 6 10:49 zfetchstats
-rw-r--r-- 1 root root 0 Jul 6 10:49 zil
-rw-r--r-- 1 root root 0 Jul 6 10:49 zstd

# Workaround

This (naive) patch sidesteps the problem:

$ diff -Naur /usr/sbin/arc_summary.old /usr/sbin/arc_summary
--- /usr/sbin/arc_summary.old 2022-07-06 10:59:50.752833101 -0400
+++ /usr/sbin/arc_summary 2022-07-06 10:59:22.449113169 -0400
@@ -255,6 +255,8 @@
     secs = SECTION_PATHS.values()

     for section in secs:
+ if not os.path.exists(PROC_PATH+section):
+ continue

         with open(PROC_PATH+section, 'r') as proc_location:
             lines = [line for line in proc_location]

# Additional information
$ lsb_release -rd
Description: Ubuntu 20.04.4 LTS
Release: 20.04
$ uname -r
5.15.0-41-generic
$ apt-cache policy zfsutils-linux linux-image-generic-hwe-20.04
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.14
  Candidate: 0.8.3-1ubuntu12.14
  Version table:
 *** 0.8.3-1ubuntu12.14 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12.9 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     0.8.3-1ubuntu12 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
linux-image-generic-hwe-20.04:
  Installed: 5.15.0.41.44~20.04.13
  Candidate: 5.15.0.41.44~20.04.13
  Version table:
 *** 5.15.0.41.44~20.04.13 400
        400 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     5.13.0.52.59~20.04.31 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     5.4.0.26.32 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Thomas Alexander Frederiksen (thomasaf)]

I can confirm that the (naive) patch sidesteps the issue, but that it is otherwise present on all our 5.15/ZFS installs.

[Alvin Cura (alvinc)]

Seeing this issue also.

$ lsb_release -rd
Description: Ubuntu 20.04.5 LTS
Release: 20.04
$ uname -a
Linux alvincura-desktop2 5.15.0-60-generic #66~20.04.1-Ubuntu SMP Wed Jan 25 09:41:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ sudo apt-cache policy zfsutils-linux
zfsutils-linux:
  Installed: 0.8.3-1ubuntu12.14
  Candidate: 0.8.3-1ubuntu12.14
  Version table:
 *** 0.8.3-1ubuntu12.14 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status
     0.8.3-1ubuntu12 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
$ sudo arc_summary -s arc
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 875, in <module>
    main()
  File "/usr/sbin/arc_summary", line 826, in main
    kstats = get_kstats()
  File "/usr/sbin/arc_summary", line 259, in get_kstats
    with open(PROC_PATH+section, 'r') as proc_location:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/spl/kstat/zfs/xuio_stats'

[Timo Aaltonen (tjaalton)]

Hello Simon, or anyone else affected,

Accepted zfs-linux into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.8.3-1ubuntu12.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Simon Déziel (sdeziel)]

I first reproduced the initial issue with zfs-linux 0.8.3-1ubuntu12.17:

$ arc_summary
Traceback (most recent call last):
  File "/usr/sbin/arc_summary", line 875, in <module>
    main()
  File "/usr/sbin/arc_summary", line 826, in main
    kstats = get_kstats()
  File "/usr/sbin/arc_summary", line 259, in get_kstats
    with open(PROC_PATH+section, 'r') as proc_location:
FileNotFoundError: [Errno 2] No such file or directory: '/proc/spl/kstat/zfs/xuio_stats'

After enabling focal-proposed, installed 0.8.3-1ubuntu12.18:

# apt-get install zfsutils-linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
   libnvpair1linux (0.8.3-1ubuntu12.18)
   libuutil1linux (0.8.3-1ubuntu12.18)
   libzfs2linux (0.8.3-1ubuntu12.18)
   libzpool2linux (0.8.3-1ubuntu12.18)
Suggested packages:
   nfs-kernel-server (1:1.3.4-2.5ubuntu3.7)
   samba-common-bin (2:4.15.13+dfsg-0ubuntu0.20.04.8)
   zfs-initramfs (0.8.3-1ubuntu12.18)
   | zfs-dracut (0.8.3-1ubuntu12.17)
Recommended packages:
   zfs-zed (0.8.3-1ubuntu12.18)
The following packages will be upgraded:
   libnvpair1linux (0.8.3-1ubuntu12.17 => 0.8.3-1ubuntu12.18)
   libuutil1linux (0.8.3-1ubuntu12.17 => 0.8.3-1ubuntu12.18)
   libzfs2linux (0.8.3-1ubuntu12.17 => 0.8.3-1ubuntu12.18)
   libzpool2linux (0.8.3-1ubuntu12.17 => 0.8.3-1ubuntu12.18)
   zfsutils-linux (0.8.3-1ubuntu12.17 => 0.8.3-1ubuntu12.18)
5 upgraded, 0 newly installed, 0 to remove and 27 not upgraded.
Need to get 1,463 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 zfsutils-linux amd64 0.8.3-1ubuntu12.18 [354 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 libuutil1linux amd64 0.8.3-1ubuntu12.18 [41.8 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 libzfs2linux amd64 0.8.3-1ubuntu12.18 [207 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 libzpool2linux amd64 0.8.3-1ubuntu12.18 [812 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 libnvpair1linux amd64 0.8.3-1ubuntu12.18 [48.4 kB]

And it now works:

$ arc_summary

------------------------------------------------------------------------
ZFS Subsystem Report Mon Aug 05 08:58:13 2024
Linux 5.15.0-117-generic 2.1.5-1ubuntu6~22.04.4
Machine: mars (x86_64) 2.1.5-1ubuntu6~22.04.4

ARC status: HEALTHY
        Memory throttle count: 0

ARC size (current): 82.7 % 1.5 GiB
        Target size (adaptive): 85.6 % 1.6 GiB
        Min size (hard limit): 6.2 % 119.3 MiB
        Max size (high water): 16:1 1.9 GiB
        Most Frequently Used (MFU) cache size: 92.1 % 1.3 GiB
        Most Recently Used (MRU) cache size: 7.9 % 116.4 MiB
        Metadata cache size (hard limit): ...