Anbox Cloud

init fails and cannot be recovered if zfsutils-linux not installed

Bug #2050009 reported by Ken Sharp
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Anbox Cloud
New
High
Unassigned
lxd
Fix Released
Unknown

Bug Description

Error: Failed to create storage pool "data": Required tool 'zpool' is missing

Installing zfsutils-linux doesn't help as it's too late.

# anbox-cloud-appliance init
Error: previous bootstrap has failed, cannot run again

If it cannot be run again then more checks need to be made that the required tools are available the first time.

2024-01-21 20:19:05 LXD network name set to lxdbr0
Since Juju 2 is being run for the first time, it has downloaded the latest public cloud information.
Only clouds with registered credentials are shown.
There are more clouds, use --all to see them.
2024-01-21 20:19:06 Public location: 192.168.7.207
2024-01-21 20:19:06 Public address: 192.168.7.207
2024-01-21 20:19:06 Private address: 192.168.7.207
2024-01-21 20:19:06 Private subnet: 192.168.4.0/22
2024-01-21 20:19:06 Using UA subscription from host
2024-01-21 20:19:06 Successfully extracted credentials from UA subscription
2024-01-21 20:19:06 Starting installation of dependencies

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Hit:1 http://192.168.4.254:3142/ubuntu jammy InRelease
Hit:2 http://192.168.4.254:3142/ubuntu jammy-updates InRelease
Hit:3 http://192.168.4.254:3142/ubuntu jammy-backports InRelease
Get:4 https://archive.anbox-cloud.io/stable jammy InRelease [2489 B]
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:6 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security InRelease
Hit:7 https://esm.ubuntu.com/apps/ubuntu jammy-apps-updates InRelease
Hit:8 https://esm.ubuntu.com/infra/ubuntu jammy-infra-security InRelease
Hit:9 https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates InRelease
Fetched 2489 B in 2s (1440 B/s)
Reading package lists...
Building dependency tree...
Reading state information...
38 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/anbox-cloud.list:1 and /etc/apt/sources.list.d/ubuntu-anbox-cloud.list:1

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-6.5.0-14-generic is already the newest version (6.5.0-14.14~22.04.1).
linux-headers-6.5.0-14-generic set to manually installed.
linux-modules-extra-6.5.0-14-generic is already the newest version (6.5.0-14.14~22.04.1).
linux-modules-extra-6.5.0-14-generic set to manually installed.
The following additional packages will be installed:
  libcpufreq0 libtcl8.6 linux-headers-5.15.0-91
  linux-headers-5.15.0-91-generic linux-image-5.15.0-91-generic
  linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic
  tcl-expect tcl8.6
Suggested packages:
  tk8.6 fdutils linux-doc | linux-source-5.15.0 linux-tools tcl-tclreadline
The following NEW packages will be installed:
  cpufrequtils expect libcpufreq0 libtcl8.6 linux-headers-5.15.0-91
  linux-headers-5.15.0-91-generic linux-headers-generic
  linux-image-5.15.0-91-generic linux-image-generic
  linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic
  tcl-expect tcl8.6
0 upgraded, 13 newly installed, 0 to remove and 38 not upgraded.
Need to get 114 MB of archives.
After this operation, 587 MB of additional disk space will be used.
Get:1 http://192.168.4.254:3142/ubuntu jammy/universe amd64 libcpufreq0 amd64 008-2build1 [12.3 kB]
Get:2 http://192.168.4.254:3142/ubuntu jammy/universe amd64 cpufrequtils amd64 008-2build1 [34.5 kB]
Get:3 http://192.168.4.254:3142/ubuntu jammy/main amd64 libtcl8.6 amd64 8.6.12+dfsg-1build1 [990 kB]
Get:4 http://192.168.4.254:3142/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB]
Get:5 http://192.168.4.254:3142/ubuntu jammy/universe amd64 tcl-expect amd64 5.45.4-2build1 [105 kB]
Get:6 http://192.168.4.254:3142/ubuntu jammy/universe amd64 expect amd64 5.45.4-2build1 [137 kB]
Get:7 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB]
Get:8 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2860 kB]
Get:9 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2342 B]
Get:10 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB]
Get:11 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB]
Get:12 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB]
Get:13 http://192.168.4.254:3142/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2488 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 114 MB in 2s (46.8 MB/s)
Selecting previously unselected package libcpufreq0.
(Reading database ... 67209 files and directories currently installed.)
Preparing to unpack .../00-libcpufreq0_008-2build1_amd64.deb ...
Unpacking libcpufreq0 (008-2build1) ...
Selecting previously unselected package cpufrequtils.
Preparing to unpack .../01-cpufrequtils_008-2build1_amd64.deb ...
Unpacking cpufrequtils (008-2build1) ...
Selecting previously unselected package libtcl8.6:amd64.
Preparing to unpack .../02-libtcl8.6_8.6.12+dfsg-1build1_amd64.deb ...
Unpacking libtcl8.6:amd64 (8.6.12+dfsg-1build1) ...
Selecting previously unselected package tcl8.6.
Preparing to unpack .../03-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ...
Unpacking tcl8.6 (8.6.12+dfsg-1build1) ...
Selecting previously unselected package tcl-expect:amd64.
Preparing to unpack .../04-tcl-expect_5.45.4-2build1_amd64.deb ...
Unpacking tcl-expect:amd64 (5.45.4-2build1) ...
Selecting previously unselected package expect.
Preparing to unpack .../05-expect_5.45.4-2build1_amd64.deb ...
Unpacking expect (5.45.4-2build1) ...
Selecting previously unselected package linux-headers-5.15.0-91.
Preparing to unpack .../06-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ...
Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ...
Selecting previously unselected package linux-headers-5.15.0-91-generic.
Preparing to unpack .../07-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ...
Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ...
Selecting previously unselected package linux-headers-generic.
Preparing to unpack .../08-linux-headers-generic_5.15.0.91.88_amd64.deb ...
Unpacking linux-headers-generic (5.15.0.91.88) ...
Selecting previously unselected package linux-modules-5.15.0-91-generic.
Preparing to unpack .../09-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ...
Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ...
Selecting previously unselected package linux-image-5.15.0-91-generic.
Preparing to unpack .../10-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ...
Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ...
Selecting previously unselected package linux-modules-extra-5.15.0-91-generic.
Preparing to unpack .../11-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ...
Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ...
Selecting previously unselected package linux-image-generic.
Preparing to unpack .../12-linux-image-generic_5.15.0.91.88_amd64.deb ...
Unpacking linux-image-generic (5.15.0.91.88) ...
Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ...
Setting up libtcl8.6:amd64 (8.6.12+dfsg-1build1) ...
Setting up libcpufreq0 (008-2build1) ...
Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ...
Setting up tcl8.6 (8.6.12+dfsg-1build1) ...
Setting up tcl-expect:amd64 (5.45.4-2build1) ...
Setting up linux-headers-generic (5.15.0.91.88) ...
Setting up cpufrequtils (008-2build1) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up expect (5.45.4-2build1) ...
Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ...
I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-91-generic
I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-91-generic
Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ...
Setting up linux-image-generic (5.15.0.91.88) ...
Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic
/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.5.0-14-generic
Found initrd image: /boot/initrd.img-6.5.0-14-generic
Found linux image: /boot/vmlinuz-5.15.0-91-generic
Found initrd image: /boot/initrd.img-5.15.0-91-generic
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 6.5.0-14-generic
NEEDRESTART-KEXP: 6.5.0-14-generic
NEEDRESTART-KSTA: 1
Packages installation done in parallel
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
vm.max_map_count = 262144
kernel.dmesg_restrict = 1
kernel.pid_max = 4194304
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv6.neigh.default.gc_thresh3 = 8192
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
fs.aio-max-nr = 524288
net.ipv4.ping_group_range = 0 2147483647
2024-01-21 20:21:45 LXD is ready, continuing with its initialization
2024-01-21 20:21:45 Using the following preseed configuration:
config:
  cluster.https_address: 192.168.7.207:8443
  core.https_address: 192.168.7.207:8443
cluster:
  enabled: true
  server_name: lxd0
networks:
- name: lxdbr0
  type: bridge
  config:
    ipv4.nat: true
    ipv4.dhcp.expiry: infinite
    ipv4.address: 240.0.0.1/16
    ipv6.address: none
profiles:
- name: default
  devices:
    root:
      path: /
      pool: data
      type: disk
    eth0:
      type: nic
      nictype: bridged
      parent: lxdbr0
storage_pools:
- name: data
  driver: zfs
  config:
    size: 32212254720
Error: Failed to create storage pool "data": Required tool 'zpool' is missing

See original description
Tags: amd64 jammy
Ken Sharp (kennybobs)
description: updated
Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Hey Ken
   In terms of the error `Error: Failed to create storage pool "data": Required tool 'zpool' is missing`, it's an error thrown from the LXD when bootstrapping the anbox cloud, regarding the discussion around this can be found [1], the requirement here is that
```
The ZFS kernel module version needs to align with one of the versions of the tooling the LXD snap bundles, ```
Which version of LXD snap do you install on your host?

BR
Gary

1. https://discuss.linuxcontainers.org/t/lxd-5-12-zfs-stopped-working-in-lxd-error-required-tool-zpool-is-missing-when-kernel-zfs-module-version-0-8/16753/6

Revision history for this message
Ken Sharp (kennybobs) wrote :

I've dived into it further (your link helped a lot here).

Identical set-up:

1. Install Ubuntu Jammy minimal server with HWE kernel
2. Install ubuntu-advantage-tools
3. pro attach
4. pro enable anbox-cloud

From here, the failing server:

$ sudo anbox-cloud-appliance init --auto

At this point the application installs LXD 5.0.

$ snap list
Name Version Rev Tracking Publisher Notes
amc 1.20.2 549 latest/stable canonical✓ -
ams-node-controller 1.20.2 289 - canonical✓ -
anbox-cloud-appliance 1.20.1 506 latest/stable canonical✓ classic
canonical-livepatch 10.7.0 246 latest/stable canonical✓ -
core 16-2.60.4 16202 latest/stable canonical✓ core
core20 20231123 2105 latest/stable canonical✓ base
core22 20231123 1033 latest/stable canonical✓ base
lxd 5.0.2-838e1b2 24322 5.0/stable canonical✓ -
snapd 2.61.1 20671 latest/stable canonical✓ snapd

On the working server:

$ sudo snap install lxd
$ sudo anbox-cloud-appliance init --auto

The default install for LXD seems to be 5.19.

$ snap list
Name Version Rev Tracking Publisher Notes
amc 1.20.2 549 latest/stable canonical✓ -
ams-node-controller 1.20.2 289 - canonical✓ -
anbox-cloud-appliance 1.20.1 506 latest/stable canonical✓ classic
canonical-livepatch 10.7.0 246 latest/stable canonical✓ -
core 16-2.60.4 16202 latest/stable canonical✓ core
core22 20231123 1033 latest/stable canonical✓ base
lxd 5.19-8635f82 26200 latest/stable canonical✓ -
snapd 2.61.1 20671 latest/stable canonical✓ snapd

So something is telling the system to install LXD 5.0, which is causing the init to fail. The reason looks to be the same reason reported in the supplied link.

The init procedure is choosing the wrong version of LXD to install.

Revision history for this message
Simon Fels (morphis) wrote :

Hey Ken,

Thanks for the summary!

If you run `pro enable anbox-cloud` it will setup the LXD snap to track 5.0/stable as we only recommend to use LTS versions at this point. Non LTS LXD versions are only supported for ~1 month so you would be in a continuous update mode. We currently do not enforce a specific version of LXD but it must be >= 5.0 (likely to become a >= 6.0 at some point).

The error you see indicates a mismatch between the ZFS tooling the LXD snap includes vs. what the version of the kernel module in the kernel you run is.

From the logs it's not explicit if you run the 5.15 or 6.5 kernel at the moment the failure occurs. The LXD 5.19 snap has newer ZFS userspace tooling but the one part of the 5.0 snap should be still compatible.

I've took a blank 22.04 VM here and installed the 6.5 HWE kernel via

$ sudo apt install --install-recommends linux-generic-hwe-22.04

Using LXD 5.0 I can replicate your issue. The problem has been fixed for LXD 5.0 already (see https://github.com/canonical/lxd-pkg-snap/pull/258) and the fix will roll out shortly. If you're fine with using LXD 5.19+, then please go ahead with that.

Changed in anbox-cloud:
assignee: nobody → Simon Fels (morphis)
importance: Undecided → High
Simon Fels (morphis)
Changed in anbox-cloud:
status: New → Incomplete
Revision history for this message
Ken Sharp (kennybobs) wrote :

Indeed I was using the 6.5 kernel in my tests.

Bug Watch Updater (bug-watch-updater)
Changed in lxd:
status: Unknown → Fix Released
Revision history for this message
Simon Fels (morphis) wrote :

Thanks Ken!

Simon Fels (morphis)
Changed in anbox-cloud:
assignee: Simon Fels (morphis) → nobody
Ken Sharp (kennybobs)
Changed in anbox-cloud:
status: Incomplete → New
Revision history for this message
Gary.Wang (gary-wzl77) wrote :

Hey Ken,

I'm reaching out to inquire about the recent change in status for this bug to 'New.' As indicated in the comment on https://bugs.launchpad.net/anbox-cloud/+bug/2050009/comments/3, the PR (https://github.com/canonical/lxd-pkg-snap/pull/258) has been backported to the 5.0 branch of LXD. The upcoming version should include the fix:

  ```
    5.0/stable: 5.0.3-d921d2e 2024-04-23 (28373) 91MB -
  ```

  So with 6.5 kernel + LXD snap from 5.0/stable, it should work out of the box.
  Could you confirm on your end too or anything I am missing?

BR
Gary

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.