Snaps appear broken on 21.04 Beta with ZFS

[Christopher]

I've attempted to use two different userland snaps (qownnotes and keepassxc) neither function. When attempting to launch I get the following error:

```
chalbersma@abdos:~$ qownnotes
cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_t3kehe//dev: No such file or directory
```

Additionally the snaps installed "by default" have a note of "broken" next to them :

```
chalbersma@abdos:~$ sudo snap list
Name Version Rev Tracking Publisher Notes
core18 1997 latest/stable canonical✓ broken
gnome-3-34-1804 66 latest/stable/… canonical✓ broken
gtk-common-themes 1514 latest/stable/… canonical✓ broken
qownnotes 21.3.9 8183 latest/stable pbek -
snap-store 518 latest/stable/… canonical✓ broken
snapd 11402 latest/stable canonical✓ broken
```

[Christopher (chalbersma-12)]

Just an update. A classic'ly confined snap (like slack) does appear to keep working.

[Paweł Stołowski (stolowski)]

Could you please provide system log (from journalctl) and output of 'snap changes'? If there is any change with error status, the please also attach the output of 'snap change <ID>' (ID is the id of the change in error). Thank you

[Christopher (chalbersma-12)]

So, testing with the qownnotes snap atm.

```
chalbersma@abdos:~$ qownnotes
cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_Ft9tYq//dev: No such file or directory
```

journalctl entries from when it runs:

```
Apr 02 11:31:54 abdos systemd[2852]: Started snap.qownnotes.qownnotes.5cd5ac28-50e1-4c4d-b015-e47a440343f1.scope.
Apr 02 11:31:54 abdos systemd[2852]: snap.qownnotes.qownnotes.5cd5ac28-50e1-4c4d-b015-e47a440343f1.scope: Succeeded.
```

And Snap change:

```
chalbersma@abdos:~$ snap changes
ID Status Spawn Ready Summary
1 Done 2 days ago, at 15:59 PDT yesterday at 12:46 PDT Initialize system state
2 Done yesterday at 12:46 PDT yesterday at 12:47 PDT Initialize device
3 Done yesterday at 14:06 PDT yesterday at 14:07 PDT Install "keepassxc" snap
4 Done yesterday at 14:08 PDT yesterday at 14:08 PDT Remove "keepassxc" snap
5 Done yesterday at 14:37 PDT yesterday at 14:38 PDT Install "qownnotes" snap
6 Done yesterday at 14:41 PDT yesterday at 14:41 PDT Remove "qownnotes" snap
7 Done yesterday at 14:42 PDT yesterday at 14:42 PDT Install "qownnotes" snap
8 Done yesterday at 14:47 PDT yesterday at 14:47 PDT Remove "qownnotes" snap
9 Done yesterday at 14:48 PDT yesterday at 14:50 PDT Install "slack" snap
10 Done today at 11:29 PDT today at 11:29 PDT Install "qownnotes" snap
```

And for 10:

```
chalbersma@abdos:~$ snap change 10
Status Spawn Ready Summary
Done today at 11:29 PDT today at 11:29 PDT Ensure prerequisites for "qownnotes" are available
Done today at 11:29 PDT today at 11:29 PDT Download snap "qownnotes" (8183) from channel "stable"
Done today at 11:29 PDT today at 11:29 PDT Fetch and check assertions for snap "qownnotes" (8183)
Done today at 11:29 PDT today at 11:29 PDT Mount snap "qownnotes" (8183)
Done today at 11:29 PDT today at 11:29 PDT Copy snap "qownnotes" data
Done today at 11:29 PDT today at 11:29 PDT Setup snap "qownnotes" (8183) security profiles
Done today at 11:29 PDT today at 11:29 PDT Make snap "qownnotes" (8183) available to the system
Done today at 11:29 PDT today at 11:29 PDT Automatically connect eligible plugs and slots of snap "qownnotes"
Done today at 11:29 PDT today at 11:29 PDT Set automatic aliases for snap "qownnotes"
Done today at 11:29 PDT today at 11:29 PDT Setup snap "qownnotes" aliases
Done today at 11:29 PDT today at 11:29 PDT Run install hook of "qownnotes" snap if present
Done today at 11:29 PDT today at 11:29 PDT Start snap "qownnotes" (8183) services
Done today at 11:29 PDT today at 11:29 PDT Run configure hook of "qownnotes" snap if present
Done today at 11:29 PDT today at 11:29 PDT Run health check of "qownnotes" snap
Done today at 11:29 PDT today at 11:29 PDT Connect qownnotes:desktop-legacy to core:desktop-legacy
Done today at 11:29 PDT today at 11:29 PDT Connect qownnotes:home to core:home
Done today at 11:29 PDT today at 11:29 PDT Connect qownnotes:network-bind to core:network-bind
Done today at 11:29 PDT today at 11:...

[Christopher (chalbersma-12)]

An update, the newest snapd seems to have the same issue:

```
chalbersma@abdos:~$ apt-cache policy snapd
snapd:
  Installed: 2.49.2+21.04ubuntu1
  Candidate: 2.49.2+21.04ubuntu1
  Version table:
 *** 2.49.2+21.04ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu hirsute/main amd64 Packages
        100 /var/lib/dpkg/status
```

[Nicholas Omann (alphacluster)]

I had this same issue on my first attempt to install 21.04 where I couldn't install any snaps or open Ubuntu Software. I found that if i reinstalled and did not use ZFS this did not happen. So I think this is a bug just with the ZFS install version.

[Christopher (chalbersma-12)]

@alphacluster I too am using ZFS (and would like to continue using it). And am willing to debug if needed.

[Christopher (chalbersma-12)]

So I found a partial workaround. Looks like filesystems aren't getting mounted (hinted at with the broken in the snap list).

I could confirm that by using this command:

```
systemctl status snap-*.mount
```

So remounting and things seem to "work":

```
systemctl restart snap-*.mount
```

I'm still running into problems with qownnotes, but that looks like an application error associated with the Wayland move that I'll have to try to fix somehow.

So the good news is that there's now a workaround.

[Christopher (chalbersma-12)]

Additionally, while the workaround let's me launch snaps, the super-majority still seem to be broken because of Wayland things.

[simon c (owslla)]

I am also experiencing the broken snaps under the default ZFS testcase (21.04 April, 4th build).

Interestingly, when using the same install media, I did not experience any issue with snaps when doing the LVM encrypted testcase.

I can also confirm the workaround posted (#7) by Christopher worked in my case.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1922293

[Christopher (chalbersma-12)]

@ubuntuqa Shiny. I'm running a few of the snaps, most don't seem to be working with Wayland (or running on an Xsession either). Is that a place to report the breakages? Just to be clear these are snaps that do seem to work on 20.04 just not 21.04 which seems to be an error with the later.

[Paweł Stołowski (stolowski)]

Thanks for reporting, reproduced, the problem is indeed strictly ZFS-related and appears to be related to a number of zfs-related services and mount units that need to be ready before mount units for snaps can be executed during boot, otherwise snap blobs under /var/lib/snapd/snaps/* cannot be accessed and mounted.

Sample log entries:

kwi 06 12:44:18 pawel-virtual-machine systemd[1]: Failed to mount Mount unit for gnome-3-34-1804, revision 66.
kwi 06 12:44:18 pawel-virtual-machine mount[803]: mount: /snap/gtk-common-themes/1514: special device /var/lib/snapd/snaps/gtk-common-themes_1514.snap does not exist.
kwi 06 12:44:18 pawel-virtual-machine mount[804]: mount: /snap/snap-store/518: special device /var/lib/snapd/snaps/snap-store_518.snap does not exist.
kwi 06 12:44:18 pawel-virtual-machine mount[802]: mount: /snap/gnome-3-34

It seems that adding "After=zfs-mount.service" to snap .mount units solves it, but the fix needs to be applied in snapd that generates them as any manual changes to these files will eventually get lost anyway (also, someone with better understanding of zfs should confirm such fix is correct).

(as for Wayland issues, let's please not mix it in here, please open a separate bug report if needed, possibly adding ubuntu-desktop to it).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Vogt (mvo)]

This looks a lot like an issue with zfs-linux to me (and *maybe* systemd, not sure about this).

The issue is that the snaps are on /var/lib/snapd/snaps and snapd writes mount units into /etc/systemd/system/ so that the snaps are mounted under /snap/<snapname>/<rev>.

Looking at a zfs-linux system and the "mount" output /var/lib is mounted:

$ mount |grep var/lib
rpool/ROOT/ubuntu_azzm5v/var/lib on /var/lib type zfs (rw,relatime,xattr,posixacl)

and the systemd unit for this mount point is auto-generated:
$ systemctl status var-lib.mount
var-lib.mount - /var/lib
     Loaded: loaded (/proc/self/mountinfo)
     Active: active (mounted) since Tue 2021-04-06 16:38:58 CEST; 19min ago
      Where: /var/lib
       What: rpool/ROOT/ubuntu_azzm5v/var/lib

Systemd should order mounts so that var-lib.mount is mounted before What=/var/lib/snapd/... is used. I see a zfs-generator in the zfs-linux source tree that looks like it should generator the mount entries from the pool. However nothing generated in /run/systemd/generator:

$ find /run/systemd/generator
/run/systemd/generator
/run/systemd/generator/netplan.stamp
/run/systemd/generator/openvpn.service.wants
/run/systemd/generator/local-fs.target.wants
/run/systemd/generator/local-fs.target.wants/systemd-remount-fs.service
/run/systemd/generator/swap.target.requires
/run/systemd/generator/swap.target.requires/dev-disk-by\x2duuid-9c43c866\x2dbe89\x2d499c\x2d9300\x2d098d5dcb0509.swap
/run/systemd/generator/dev-disk-by\x2duuid-9c43c866\x2dbe89\x2d499c\x2d9300\x2d098d5dcb0509.swap
/run/systemd/generator/boot-grub.mount
/run/systemd/generator/local-fs.target.requires
/run/systemd/generator/local-fs.target.requires/boot-grub.mount
/run/systemd/generator/local-fs.target.requires/boot-efi.mount
/run/systemd/generator/boot-efi.mount

We could easily workaround this via Pawels approach above but I feel this will affect more than snapd. Everyone who wants to mount anyting under /var/lib will be affected.

[Michael Vogt (mvo)]

[from-irc]
<didrocks> mvo: hey! There are 2 things here:
<didrocks> - the zfs generator is currently broken in the default setup due to OpenZFS 2.0 default layout. ubiquity in unapproved since this morning is fixing this
<didrocks> (people using our default setup will not be impacted by the bug then)
<didrocks> - then, for people using ZFS themselves (no ZSys/not our default setup with a generator), indeed, all the .mount units needs to specify After=zfs-mount.service as
<didrocks> Pawel suggested
<didrocks> (as the mount points are only available once zfs-mount.service has run)
<didrocks> mvo: I guess cking (who is our ZFS maintainer) can have a second eye on this, but the truth shouldn’t be far from that :)
<mvo> didrocks: cool, thanks. so your recommendation is that we implement the workaround?
<mvo> didrocks: for those who use non-defaults?
<didrocks> mvo: correct, for everyone using non default having a dataset as a parent directory where you mount the snaps (for the .mount files)

[Christopher (chalbersma-12)]

> (as for Wayland issues, let's please not mix it in here, please open a separate bug report if needed, possibly adding ubuntu-desktop to it).

@stolowski No worries. I'll open a new one for that.

@mvo Let me know when a new release is cut and I can pop it on the box and test.

[Luna Jernberg (bittinubuntu)]

Reinstalled my desktop today, with todays Beta and having the same issue :(

[Paweł Stołowski (stolowski)]

A tentative workaround for snapd discussed above: https://github.com/snapcore/snapd/pull/10113

[Colin Ian King (colin-king)]

I'm still trying to figure out where/why/how this is a ZFS issue, but looking at the workaround I think it's a good way forward.

The new line added "After=zfs-mount.service" - how does this affect things if ZFS is not installed?

[Paweł Stołowski (stolowski)]

@Colin After=.. only defines ordering but it's a loose coupling and has no effect is the service doesn't exist.

[Dimitri John Ledkov (xnox)]

somehow i was expecting the snap mount units to start after local-fs.target, and for zfs.target to complete by then. But it looks like that's not how things are ordered unfortunately.

[Luna Jernberg (bittinubuntu)]

This is fixed in todays build, so closing this bug

[Paweł Stołowski (stolowski)]

@Luna what was the fix, can you link to the respective change? I'm still seeing the problem with up-to-date 21.04 (and there were no updates reported for zfs-linux; was it actually released)?

The tentative fix for snapd hasn't been commited yet in snapd; if the problem was fixed elsewhere then we won't need snapd fix; therefore I'm marking this back to "in progress" in snapd and it should be set "invalid" if not needed in snapd.

[Christopher (chalbersma-12)]

Just an update, I still see this in the beta after a reboot. Workaround still functions.

[Paweł Stołowski (stolowski)]

@Christopher as far as I understand the fix happened in ubiquity installer, so it will only take effect on fresh installation (which is fine given that this is during beta). See the comments under https://github.com/snapcore/snapd/pull/10113