zfs-dkms 2.1.5-1ubuntu6~22.04.2- Kernel module failed to build

[Mike Ferreira]

On todays updates, zfs-dkms 2.1.5-1ubuntu6~22.04.2 failed to build kernel module.

Genrated Crash report, but couldn't send it on it's own.

Error it kept throwing was:
>>>
Setting up zfs-dkms (2.1.5-1ubuntu6~22.04.2) ...
Loading new zfs-2.1.5 DKMS files...
Building for 6.2.0-36-generic 6.2.0-37-generic
Building initial module for 6.2.0-36-generic
configure: error:
 *** None of the expected "iops->get_acl()" interfaces were detected.
 *** This may be because your kernel version is newer than what is
 *** supported, or you are using a patched custom kernel with
 *** incompatible modifications.
 ***
 *** ZFS Version: zfs-2.1.5-1ubuntu6~22.04.2
 *** Compatible Kernels: 3.10 - 5.19

ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/zfs-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.2.0-36-generic (x86_64)
Consult /var/lib/dkms/zfs/2.1.5/build/make.log for more information.
dpkg: error processing package zfs-dkms (--configure):
 installed zfs-dkms package post-installation script subprocess returned error exit status 10
>>>

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: zfs-dkms 2.1.5-1ubuntu6~22.04.2
ProcVersionSignature: Ubuntu 6.2.0-36.37~22.04.1-generic 6.2.16
Uname: Linux 6.2.0-36-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: GNOME
Date: Sat Nov 25 21:00:27 2023
InstallationDate: Installed on 2021-09-23 (793 days ago)
InstallationMedia: Ubuntu 20.04.3 LTS "Focal Fossa" - Release amd64 (20210819)
PackageArchitecture: all
SourcePackage: zfs-linux
UpgradeStatus: Upgraded to jammy on 2022-08-17 (466 days ago)

To recreate:
- Install Ubuntu 22.04.3.
- Update to current.
- Install package 'zfs-dkms'

[Mike Ferreira (mafoelffen)]

There weer no other make logs for any other kernels present(???)
>>>
mafoelffen@Mikes-ThinkPad-T520:~/Downloads$ ls /var/lib/dkms/zfs/2.1.5/
5.15.0-89-generic source
>>>

[Mike Ferreira (mafoelffen)]

I remember this error on zfs-dkms when the kernel ranges went from 5.19 to 6.2.0-X... With package zfs-dkms 2.1.5-1ubuntu6~22.04.1.

RE: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2030100

That was fixed a few updates after that, but now we are back to this:
>>>
 *** ZFS Version: zfs-2.1.5-1ubuntu6~22.04.2
 *** Compatible Kernels: 3.10 - 5.19
>>>
Which makes me think that the patch that was applied for that last package, was not applied to this new one.

*** Should I post a warning on the Forum in Section "Installations & Updates", for ZFS users not to apply updates until this is resolved? I've already sent a few PM's out to members I know this will affect...

[Mike Ferreira (mafoelffen)]

This is not a problem for Noble, as it uses
>>>
fs-dkms/noble,noble 2.2.0-0ubuntu3 all
>>>
Not a problem for Mantic, as it uses:
>>>
zfs-dkms/mantic,mantic 2.2.0~rc3-0ubuntu4 all
>>>

BUT-- In today's updates for Lunar:
>>>
libzfs4linux/lunar-updates,lunar-security 2.1.9-2ubuntu1.2 amd64 [upgradable from: 2.1.9-2ubuntu1.1]
zfs-dkms/lunar-updates,lunar-updates,lunar-security,lunar-security 2.1.9-2ubuntu1.2 all [upgradable from: 2.1.9-2ubuntu1.1]
zfs-initramfs/lunar-updates,lunar-security 2.1.9-2ubuntu1.2 amd64 [upgradable from: 2.1.9-2ubuntu1.1]
zfs-zed/lunar-updates,lunar-security 2.1.9-2ubuntu1.2 amd64 [upgradable from: 2.1.9-2ubuntu1.1]
zfsutils-linux/lunar-updates,lunar-security 2.1.9-2ubuntu1.2 amd64 [upgradable from: 2.1.9-2ubuntu1.1]
>>>
I will clone that test machine and see if it will crash on Lunar with that update... BRB.

[Mike Ferreira (mafoelffen)]

No. Lunar upgraded fine to that package version with the same kernel.

But:
on this 22.04.3 machine now:
>>>
mafoelffen@Mikes-B460M:/data/ISO$ apt list zfs-dkms
Listing... Done
zfs-dkms/jammy-updates,jammy-updates,jammy-security,jammy-security,now 2.1.5-1ubuntu6~22.04.2 all [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
mafoelffen@Mikes-B460M:/data/ISO$ sudo apt install --reinstall zfs-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 25 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for zfs-dkms:amd64
>>>
Look at the error on the end...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rick S (1fallen)]

I have no logs, it blew my kernel out of boot.
"you need to load the kernel first"

[Mike Ferreira (mafoelffen)]

Attempt at remove and reinstall of package zfs-dkms failed (see attached)

No, Crashed again on zks-dkms. Can't et past it. And only affects Jammy.

[Mike Ferreira (mafoelffen)]

Posted a warning out on the Forum: https://ubuntuforums.org/showthread.php?t=2492886&p=14166863#post14166863

[Jimmmy Mick (jimmmymick)]

This bug prevents successful installation of any other package, so I think its importance should be upgraded.

[Mike Ferreira (mafoelffen)]

Seems to also affect Mantic, if it was installed as jammy, then upgraded to Mantic. But doesn't seem to affect Mantic systems if they were installed fresh as Mantic. All Tests, so far, show it does not affect Noble at all (either upgraded from Mantic or installed fresh).

I think it is isolated this version of the zfs-dkms package. Last time this was a problem, it was because of the supported kernel range set in the code of zfs-dkms. When HWE went from 5.19 to 6.2, it failed. It got patched. I don't think this version included that patch, especially when the error throws an unsupported kernel version error.

The module is built by zfsuitls-linux... "I think" a short term work-around might be to temporarily uninstall zfs-dkms, and watch very carefully for any kernel version updates. If there were a kernel update, then before the next reboot, just reinstalling zfsutils-linux should build the new module for that kernel.

*** I haven't tested that yet... Will do a test case on that in the morning.

I have 3 other systems 'here' that are 22.04 LTS, ZFS-On-Root... I paused the updates on those 3 other systems until this gets resolved.

[Mike Ferreira (mafoelffen)]

It is not just preventing upgrades, it is preventing the upgrade to Linux kernel 6.2.0-37-generic
>>>
mafoelffen@Mikes-ThinkPad-T520:~$ modinfo zfs | grep -e '^filename\|^version'
filename: /lib/modules/6.2.0-36-generic/kernel/zfs/zfs.ko
version: 2.1.9-2ubuntu1.1
mafoelffen@Mikes-ThinkPad-T520:~$ uname -r
6.2.0-36-generic
mafoelffen@Mikes-ThinkPad-T520:~$ ls /boot/ | grep -e 'initrd\.img-[5-6]\|vmlinuz-[5-6]'
initrd.img-5.15.0-89-generic
initrd.img-6.2.0-36-generic
initrd.img-6.2.0-37-generic
vmlinuz-5.15.0-89-generic
vmlinuz-6.2.0-36-generic
vmlinuz-6.2.0-37-generic
>>>

[Mike Ferreira (mafoelffen)]

I got tired of waiting for anyone to triage this... I removed package zfs-dkms so I could applied my updates...

They upgraded successfully. Including building the modules for kernel 6.2.0-37-generic.

Then installed package zfs-dkms, where it failed again on building the module for any 6.2.0-xx kernels.

[Mike Ferreira (mafoelffen)]

That blew out the ability to boot. Ended up having to remove zfs-dkms. Reinstalling kernel and boot.

Back up.

zfs-dkms is broken big time for 22.04.

[Walter Tautz (wtautz)]

Just to confirm I have the same problem on 22.04 with kernel:

Building initial module for 6.2.0-37-generic
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<stdin>:4: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<stdin>:4: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<stdin>:3: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<stdin>:3: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<stdin>:2: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<stdin>:2: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
<string>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<string>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
configure: error:
 *** None of the expected "iops->get_acl()" interfaces were detected.
 *** This may be because your kernel version is newer than what is
 *** supported, or you are using a patched custom kernel with
 *** incompatible modifications.
 ***
 *** ZFS Version: zfs-2.1.5-1ubuntu6~22.04.2
 *** Compatible Kernels: 3.10 - 5.19

ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/zfs-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.2.0-37-generic (x86_64)
Consult /var/lib/dkms/zfs/2.1.5/build/make.log for more information.
dpkg: error processing package zfs-dkms (--configure):
 installed zfs-dkms package post-installation script subprocess returned error exit status 10

[Walter Tautz (wtautz)]

I suppose the answer is to edit the postinst script albeit it looking for some kind of compile time
function when trying to build the kernel module.

[Mike Ferreira (mafoelffen)]

Before doing anything, backup your bpool and rpool via rsync so you have a fall back point.

Start a thread on UbuntuForums > Installations and Updates. I will talk you through a work-around to uninstall zf-dkms and everything else I did to get around it. It's too long a process for this bug report, There was lessons learned by both I and 1fallen, where both ours crashed and we had to recover from that.

Like I said. Get good backups now, While it's still broken, but still bootable.

[Adrian Moldovan (adi-mol)]

A bit surprised that this isn't treated with more urgency. We're on an LTS ubuntu and we get broken updates, basically.

Though so far I can't see it having a major impact in my case (can still use zfs and updating other packages seems to work)

[Mike Ferreira (mafoelffen)]

<<Yes, I am surprised no-one has even triaged this.>>

Rick S and I have talked a lot today... trying to remember:

I added a repo on my GitHub on: Open-ZFS-Admin > "How To Uninstall package 'zfs-dkms' And Survive"

We had to remember:

- Why we actually started using it.

Ubuntu, in the past, did not rebuild the zfs module, so we needed zfs-dkms so that when kernels were updated, that the 'zfs' modules were built for the new kernel...

- If it is actually needed anymore

Not for normal users... I will explain that. I think sometime around 22.04.x, canonical started adding the zfs modules in-kernel. Confirmed that, if you install Ubuntu 22.04.3 as ZFS-On_Root, and do updates, zfs-dkms is not an installed package, and the 'zfs' modules are built during a kernel update. So it is not actually needed any longer by normal users.

- Why Rick S (1fallen) and I still need to use package 'zfs-dkms', if it is not required as a default package anymore...

We do DEV testing and ZFS version (zfs-linux) testng/validations... We need it becaseu we often install and use kernels from the Ubuntu mainline repo. Which does not have the added Ubuntu mod's and additions. That, and (I hate to say this) sometimes Canonical forgets to add things, and has regression issues.

This package has saved our behinds sometimes.

I am embarrassed at Launchpad's participation with this. i will take the baton and help them with that.

When I get that done, and Rick and I get that tested, I will publish the link to that here,.. When I get those instructions finished, and everything is tested so I can confirm it works, with just what needs to be done.

[Mike Ferreira (mafoelffen)]

I've tested it and handed it off to Rick for him to look it over and test (second set of eyes)...

[Mike Ferreira (mafoelffen)]

Here is the link:
https://github.com/Mafoelffen1/OpenZFS-Ubuntu-Admin/blob/main/ZFS-DKMS-WORK-AROUND.md

It is tested. It is somewhat daunting... But it works.

[LaurentC (laurent-chardon)]

Thank you @mafoelffen, the workaround worked perfectly.

[adiDev (adidev7)]

I'm frightened to reboot the system now. The status after apt upgrade:

```
Setting up zfs-dkms (2.1.5-1ubuntu6~22.04.2) ...
Removing old zfs-2.1.5 DKMS files...
Deleting module zfs-2.1.5 completely from the DKMS tree.
Loading new zfs-2.1.5 DKMS files...
Building for 6.2.0-35-generic 6.2.0-37-generic
Building initial module for 6.2.0-35-generic
configure: error:
        *** None of the expected "iops->get_acl()" interfaces were detected.
        *** This may be because your kernel version is newer than what is
        *** supported, or you are using a patched custom kernel with
        *** incompatible modifications.
        ***
        *** ZFS Version: zfs-2.1.5-1ubuntu6~22.04.2
        *** Compatible Kernels: 3.10 - 5.19

ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/zfs-dkms.0.crash'
Error! Bad return status for module build on kernel: 6.2.0-35-generic (x86_64)
Consult /var/lib/dkms/zfs/2.1.5/build/make.log for more information.
dpkg: error processing package zfs-dkms (--configure):
 installed zfs-dkms package post-installation script subprocess returned error exit status 10
Processing triggers for initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: Generating /boot/initrd.img-6.2.0-37-generic
Errors were encountered while processing:
 zfs-dkms
E: Sub-process /usr/bin/dpkg returned an error code (1)
```

Result from dpkg -s zfs-dkms:

```
Package: zfs-dkms
Status: install ok half-configured
Priority: optional
Section: contrib/kernel
Installed-Size: 17606
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Source: zfs-linux
Version: 2.1.5-1ubuntu6~22.04.2
Config-Version: 2.1.5-1ubuntu6~22.04.1
Replaces: spl-dkms
Provides: zfs-modules
Depends: dkms (>> 2.1.1.2-5), file, libc6-dev | libc-dev, lsb-release, python3-distutils | libpython3-stdlib (<< 3.6.4), debconf (>= 0.5) | debconf-2.0, perl:any
Recommends: zfs-zed, zfsutils-linux (>= 2.1.5-1ubuntu6~22.04.2), linux-libc-dev (<< 5.20~), linux-libc-dev (>= 3.10~)
Suggests: debhelper
Breaks: spl-dkms (<< 0.8.0~rc1)
Description: OpenZFS filesystem kernel modules for Linux
 OpenZFS is a storage platform that encompasses the functionality of
 traditional filesystems and volume managers. It supports data checksums,
 compression, encryption, snapshots, and more.
 .
 This DKMS package includes the SPA, DMU, ZVOL, and ZPL components of
 OpenZFS.
Homepage: https://zfsonlinux.org/
Original-Maintainer: Debian ZFS on Linux maintainers <email address hidden>
```

[Mike Ferreira (mafoelffen)]

From what I got after that error (originally)... It errors in dpkg, so it leaves the old modules built for the kernels that it did upgrade to successfully (there).

Mine error'ed out when it tried to upgrade from 6.2.0-36 to 6.2.0-37. If worried me also, BUT... It did reboot.

Since linux-image-6.2.0-37-generic errored out during the install, it does not add it to the Grub2 grub.cfg menu, so on reboot, it booted into 6.2.0-36 as the running kernel, on the modules it built.

Each time it tried to do anything (updates), it would install anything new manually, but not actually do any of the updates. It looked like they were updating, then hit that zfs-dkms error.

From the output, it initially looked like it installed all the update except zfs-dkms, but that assumption was wrong. Next time I tried to update, it tried to update those same exact packages that were in that previous update (grouping), plsu anything newly added to those-- which tells me it did not upgrade any of them.

This morning, I see that Mantic and Noble got updates for zfs-dkms... But I haven't heard yet about Lunar. I don't see anything in any of my Jammy machine's updates.

The work-around I came up with... I haven't found an easier way. If you remove it before that update failure, then you do not need to reinstall linux-headers-XXX and update the instramfs files... Because it didn't hit the bug yet where it failed to build the modules. BUT...

And this is a big "BUT", something with zfs-dkms is somewhat 'evil'. When you remove it, somehow it exports all the zpools besides rpool!!!

*** It does not do that immediately. ***

Somehow, between the shutdown and reboot it exports all the pools besides rpool. I don't know why, but it just does. (That is why I refered to zfs-dkms' as soemhow being 'evil'. It has deep roots.)

While still booted, before that, everything is still mounted. You can export and import the pools. It does not matter. On the next boot, after removing package 'zfs-dkms', nothing is mounted except for rpool.

That work-around has instructions on how to import/mount everything back up and how to get it booting again.

*** Like I found out, package 'zfs-linux' is not needed anymore to build the zfs module during a kernel update. That was added by Canonical in-kernel. No one announced that change when that happened. We (Us Users) know that now.

Only a few of us still need zfs-dkms for out-of-series kernels, and testing DEV and Proposed packages, where weird circumstances are presented. But we are doing things that are "not normal", for the sake of Ubuntu and it's users.

[Mike Ferreira (mafoelffen)]

Just passed on to me from FL140: https://github.com/openzfs/zfs/issues/15586#issuecomment-1836806381

Is an updated script to manually build and install the OpenZFS native deb packages, including 'zfs-dkms' 2.2.2.

[Jimmmy Mick (jimmmymick)]

I just got prompted to update to 23.04 / Lunar - is that safe to attempt while suffering the effects of this bug? Alternatively, can I try an update to 23.10 / Mantic?

[Mike Ferreira (mafoelffen)]

@jimmymmick -- From LTS to a dev interim version? Maybe if other things supported newer hardware, but I am on 13th Gen iCore and still doing fine on 22.04.3. The HWE stack is the Lunar kernels (6.2.0-x). But Lunar loses support soon any ways...

In fact if you update to Mantic (23.10) right now, it will flat not work. I'm testing zfs-linux 2.2.2 right now, and got it pushed through to Noble Main.

Mantic is affected to, if you created your modules from earlier. Rick S above, was Mantic and affected.

zfsutils-linux 2.2.2 breaks zfs-dkms 2.2.2, so that is not going to work. (I know need to file separate bug on that.)

I would recommend using my work-around from comment #24. I just got all machines out fro under zfs-dkms now. zfs-dkms is not longer needed by 97% of users now.

If for some reason you do need an out-of-series Linux kernel, then rebuild the modules for that kernel... Those instructions are in comment #28. Exclude zfs-dkms from that. I have one machine here that I test new kernels on.

[J C Nash (nashjc)]

My issue with the zfs-dkms bug was raised by Linux Mint update giving an error message each time
an upgrade was run. The error message has now stopped appearing. I've not seen any news that the
issue is fixed. Does anyone have information that the corrections have migrated through from the
upstream repos? JN

[J C Nash (nashjc)]

As far as I can tell, this has now resolved for me. Anyone still having issues?

[Mark Weisser (wriver4)]

Well, I have the same problem as the original post except I am running, but I can't get rid of the trying to install loop. It fails to reinstall when updating other packages.

zfs version
zfs-2.1.5-1ubuntu6~22.04.2
zfs-kmod-2.2.0-0ubuntu1~23.10

[Richard Harris (technomagus)]

Same here: can't appear to install anything new

zfs-2.1.5-1ubuntu6~22.04.4
zfs-kmod-2.2.0-0ubuntu1~23.10.2

I'm not seeing any time-to-fix notices