When using HWE, zfs-kmod and zfs user tools versions must match

[Tim K.]

Ubuntu 20.04.2 LTS

I ran into a problem recently (https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1939177) due to mismatched versions when HWE upgraded the kernel from 5.8 to 5.11 where the userspace tools don't match the zfs-kmod version. After the update the system ended up with zfs-0.8.3-1ubuntu12.12 and zfs-kmod-2.0.2-1ubuntu5

My understanding from asking in the OpenZFS github is that the two versions must match.

This bug is to figure out what to do for those of us who use HWE and zfs.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Petersen (rambus666)]

I have a proposal to what could be done to fix this.
And i hope the origional creater of this bug agree.
As well as other in this thread.

But if we have a HWE kernel that gives us the newest stable kernel.

Wouldten it be an idea to have a zfsutils-linux-hwe package that matches the ZFS kmod in the kernel.

Depending on which kernel a person uses.
It would be as simple as either installing

zfsutils-linux

or

zfsutils-linux-hwe

----

Best regards,
Darkyere

[Marcus Schmidt (marcus.schmidt)]

Hi,

any updates on this ?

Thanks.

[James Dingwall (a-james-launchpad)]

The version mismatch was the cause of a problem I had with `zfs promote`: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1947568

In the end I'm using a 3rd party PPA to get updated user space tools but I don't see that as the ideal solution.

[Galinette (galinette34)]

I can confirm problems relating to this ticket while transferring ZFS clones with zfs send and receive. This impacts on most Docker and LXD installations using ZFS, since the container images are saved on zfs as chained clones (each clone relating to another) .

Reproduced on:

Operating System: Ubuntu 20.04.3 LTS
Kernel (HWE): Linux 5.11.0-40-generic
ZFS Packages: 0.8.3-1ubuntu12.13
ZFS Kernel Module: 2.0.2-1ubuntu5.4

The problem can be reproduced with the attached script (zfsbug_with_hwe.tar.gz).
The error is sporadic so you have to start the script several times until the following occurs:
 "cannot receive: local origin for clone dummypool2/dataset1/clone2@snap3 does not exist"

[Galinette (galinette34)]

Script helping to reproduce problems relating to this ticket while transferring ZFS clones with zfs send and receive.

[N. X. (n1k7a5)]

I feel like this is a subject that needs to be resolved.

Recently when I tried using zfs send and receive from ubuntu to a fresh install of ubuntu 20.04.3 LTS.
I could 'ten receive the dataset properly.

After looking at the net I discovered that the ZFS kmod and User tools didn’t match.
Because this fresh install of Ubuntu 20.04.3 had HWE kernel by default and that meant the ZFS kmod and user tools In the LTS repository didn’t match.

According to OpenZFS github this is a very bad thing.

Basically when one chooses an Ubuntu LTS. Its because it is the stable version of ubuntu with no breaking changes.
Making HWE kernel be the default does the exact opposite of this and actually makes a breaking change to ZFS and that is exactly what is not supposed to happen.

But for some reason canonical is overlooking this on ZFS where people store there important files.

This is still a breaking change and in the future it could potentially lead to data loss which would cause a huge uproar when people start loosing their files because a new kernel had a kmod that didn’t match the user space tools in the repositories.

I really feel like canonical should have a look at this and understand this does exactly what it is not supposed to do on an LTS. It gives breaking changes by having the HWE kernel pr. default and not having the option to use a matching ZFS user tools In their repo.

In the end I manually had to install Linux-image-generic and accompanying modules, modules-extra and headers packages.
Then remove anything related to HWE kernel to make sure I never booted on the wrong kernel with the mismatching kmod/userspace of zfs.

And after that I had no ZFS send/receive problems

I would just wish I could get a corresponding zfsutils-linux that matches the HWE kernel so that I can stay on the latest kernel without breaking ZFS.

[Mark Petersen (rambus666)]

I see a lot of people somehow subscribed to this bug.

But only 17 registered as this bug affects them.

Could i politely ask more people to vote for this issue.

So it might be noticed before 22.04.
Since we need a fix before then or we will live with this for 2 more years in the new LTS.

Best regards,
Darkyere

[Some Guy On The Net (lockszmith)]

This affects https://bugs.launchpad.net/ubuntu/+source/zsys/+bug/index.php?1959017 (for some reason couldn't link the bug URL)

GitHub URL is: https://github.com/ubuntu/zsys/issues/220

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[d (agentefilemonpi)]

I have the same problem. In my case, it is caused by upgrading the NVIDIA driver metapackage to 510 from 470.

I am very grateful to @lockszmith for sharing his steps on https://askubuntu.com/questions/1388997/zsys-commit-service-fails-with-couldnt-promote-dataset-not-a-cloned-filesy

It really makes me very wary of upgrading my systems. Those are LTS releases and things like this are very concerning.