shiftfs-lxd.sh in ubuntu_overlayfs_smoke_test failed with "Required tool 'zpool' is missing"

[Po-Hsu Lin]

Issue found in cycle 2024.04.01 with F-gkeop (5.4.0-1091.95), F-oracle (5.4.0-1123.132)

Test log
  Running 'sudo -iu ubuntu /home/ubuntu/autotest/client/tmp/ubuntu_overlayfs_smoke_test/src/overlay-shiftfs-tests/tests/shiftfs-lxd.sh'
  Error: Required tool 'zpool' is missing
  Creating c1
  Error: Failed loading storage pool "zfs": Storage pool not found
  Error: Failed to fetch instance "c1" in project "default": Instance not found
  FAIL
  Error: Failed to fetch instance "c1" in project "default": Instance not found
  Error: Failed checking instance exists "local:c1": Failed to fetch instance "c1" in project "default": Instance not found
  Error: Storage pool not found

This is the failing command in shiftfs-lxd.sh:
  $ sudo lxc storage create zfs zfs size=2GiB
  Error: Required tool 'zpool' is missing

$ snap list lxd
Name Version Rev Tracking Publisher Notes
lxd 5.21.1-43998c6 28155 latest/stable canonical✓ -

This issue does not exist in 2024.03.04 cycle for these two kernels.

[Po-Hsu Lin (cypressyew)]

I tried the same command with Focal 5.4.0-1089-gkeop in 2024.03.04 cycle, and it's failing with the same issue.

$ sudo lxc storage create zfs zfs size=2GiB
Error: Required tool 'zpool' is missing
$ uname -a
Linux f-lgkeop-gkeop-5-4-e2hcpu8-u-overlayfs-smk-test 5.4.0-1089-gkeop #93-Ubuntu SMP Wed Mar 13 18:13:26 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

This looks like a regression in LXD to me.

[Po-Hsu Lin (cypressyew)]

Comment from Simon Déziel:
5.21 and later require ZFS 2.1 or later. This means if you are using Ubuntu 20.04, you need the HWE kernel.

So this failure is expected. We should either hint it or skip it from the test.

[Po-Hsu Lin (cypressyew)]

LXD release note:
https://discourse.ubuntu.com/t/lxd-5-21-1-lts-has-been-released/43823/5

[Po-Hsu Lin (cypressyew)]

LXD team got this issue resolved by pushing a version that supports ZFS 0.8 to 5.21/stable channel.

Please find the detailed comment in the discourse post above.

I have the MP updated.

[Po-Hsu Lin (cypressyew)]

Patch applied, failed test restarted.