(Encrypted) ZFS breaks 22.04 installation

[Nils Büchner]

On a freshly installed 22.04.0 amd64 Desktop with (encrypted) ZFS, removing the Firefox snap breaks:
* the current gnome-shell session - it returns to gdm3 login prompt
* snapd - all snaps are marked 'broken'
* dpkg - dpkg status database is lost

Reproducible on dedicated hardware and VMs. Not reproducible on a default (/ on ext4) installation.

Video showing the whole process on a freshly installed system:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1970066/+attachment/5583102/+files/1970066.webm

journalctl -b output:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1970066/+attachment/5583100/+files/1970066_journal.txt

/proc/self/mountinfo:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1970066/+attachment/5583101/+files/1970066_mountinfo.txt

Installation procedure:
- Minimal installation
- Tick 'download updates'
- Tick 'install third party software'
- Erase disk and use ZFS + encrypt
- Require password to log in

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Erich Eickmeyer (eeickmeyer)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1970066

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Nils Büchner (n.buechner)]

user@pc:~$ apport-collect 1970066
Package snapd not installed and no hook available, ignoring

[Steffen (sldayo)]

I was the one who originally had this issue and tried to get help on IRC, so I can confirm that this happens on my system as well with ZFS (encrypted) when running "snap remove firefox" during the first boot.

I decided to reproduce it again in order to submit my information but "apport-collect 1970066" suggested that I create a separate report.

When I then ran "apport-bug snapd" it failed (see attached screenshot) with a FileNotFoundError.

When I then tried to let apport-collect send my info anyway then it also failed:

Traceback (most recent call last):
  File "/usr/share/apport/apport-gtk", line 597, in <module>
    app.run_argv()
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 719, in run_argv
    return self.run_update_report()
  File "/usr/lib/python3/dist-packages/apport/ui.py", line 604, in run_update_report
    apport.packaging.get_version(p)
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 163, in get_version
    pkg = self._apt_pkg(package)
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 156, in _apt_pkg
    return self._cache()[package]
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 121, in _cache
    self._apt_cache = apt.Cache(progress, rootdir='/')
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 143, in __init__
    self._check_and_create_required_dirs(rootdir)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 186, in _check_and_create_required_dirs
    open(rootdir + f, "w").close()
PermissionError: [Errno 13] Permission denied: '//var/lib/dpkg/status'

Edit: Regarding this error, I got the same result as in #3 when running apport-collect as root.

[Steffen (sldayo)]

I have found that installing a snap (e.g. chromium) on the first boot without removing a snap (firefox) first also bricks the system, leaving the user without any working web browser since firefox also stops working.

[Steve Langasek (vorlon)]

As root, what does ls -la /var/lib/dpkg show?

[Nils Büchner (n.buechner)]

user@pc:~$ ls -la /var/lib/dpkg
total 18
drwxr-xr-x 2 root root 2 Apr 24 08:36 .
drwxr-xr-x 66 root root 68 Apr 24 08:45 ..

[Tom Reynolds (tomreyn)]

Changing status to 'new' (from 'incomplete') since apport is broken.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bradford Powell (bradford-powell)]

From what I can tell, this critical bug is still present in the release candidate for 22.04.1 (at least, I just tried an install in a VM with zfs encryption and after running `snap remove firefox` still get pushed back to the gdm prompt and somehow /var/lib/apt ends up not being mounted after that point.

In the 3 months since the initial release and reporting of this bug, has anyone figured out with more specificity why this happens, and what the path for addressing the issue might be?

[Andrew J. Caines (cainesaj)]

Appears to be present in a vanilla Ubuntu 22.04.1 Desktop minimal install on amd64 with ZFS and encryption.

Ubuntu Software fails to launch, with errors including

  snap.snapd-desktop-integration.service: Failed at step CHDIR spawwning /usr/bin/snap: No such file or directory
  internal error, please report: running "snap-store.ubuntu-software" failed: cannot find installed snap "snap store" at revision 582: missing file /snap/snap-store/582/meta/snap.yaml

/snap/snap-store/582/ is empty. snapd.service is active (running)

"snap list" shows all snaps as "broken".

"sudo snap refresh" produces errors due to firefox's missing snap.yaml.

[Rufus Filigrane (ruffiller)]

I have a related error after a fresh install of xubuntu 22.04.1 with zfs + encryption:

After doing a first "apt update" followed by an "apt install" of a package, apt and the whole X session crash and apt is broken from then on.

Steps to reproduce:

1. Install (x)ubuntu 22.04.1 with zfs + encryption and boot installed system for the first time
2. apt update
3. apt install <some package>
→ apt and x session crash

It seems that the apt install triggers systemd to unmount every zfs mount because systemd mistakenly thinks the rpool key is not loaded. It fails unmounting /var/lib or / but succeeds with /var/lib/apt, /var/lib/dpkg. This causes apt to not work anymore.

I observed the following:

• Somehow after the first boot of the freshly installed system, the service unit /run/systemd/generator/zfs-load-key-rpool.service is not generated by zfs-mount-generator
• Because this unit must be available and active, this seems to cause systemd to think that the key for the rpool is not loaded → systemd starts unmounting all zfs mounts after some trigger

Workaround: Reboot immediately after the first boot → service unit gets generated and started correctly. This prevents any of the crashes I observed.

[Dimitri John Ledkov (xnox)]

This still sounds like a duplicate of https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1993318 which was resolved in ubiquity 23.04.1 & 22.04.19

Specifically, systems installed with zfs using ubiquity older than 22.04.19 on jammy; and older than 23.04.1 on any later release, would produce very confusing first boot state.

Everything seems to work. Until anybody calls `systemctl daemon-reload` at which point systemd would loose its mind, fail to run generators to create correct zfs mount units, and start unmounting everything it can including critical mountpoints like /home /var /usr, and so on.

If after first boot, one doesn't do anything, and reboots, everything is fine.

If during first boot one installs any snap, or any apt package that has any systemd unit, as part of the process `systemctl daemon-reload` will be executed, and trash the system.

Reboot however, will recover the system and everything should be working as normal.

I would want to mark this as duplicated of https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1993318 unless anybody can reproduce any bad behaviour after install with fresh enough daily isos of jammy or mantic, and try to "do lots of actions that could break system" _during_ the first boot after the install.

[Brian Murray (brian-murray)]

I was unable to recreate this after first boot of an install of `Ubuntu 22.04.3 LTS "Jammy Jellyfish" - Release amd64 (20230807.2)`.

I successfully installed a deb, 2vcard after and apt-get update, and a snap - joplin.