snapd doesn't regenerate existing mount units when mountunit template changes

[Aleksandar Ivanisevic]

I have a cluster that is supposed to have two identical members. However I've spotted inconsistencies although the versions of snap and snap core are the same. The only difference that I was able to spot is the order of installation, note that on qbs01 core18 was installed before core and on qbs02 it is the other way round:

support@qbs01:~$ diff -u <(cat /etc/systemd/system/snap-core18-2066.mount) <(ssh qbs02 cat /etc/systemd/system/snap-core18-2066.mount)
--- /dev/fd/63 2021-05-19 20:58:22.400729632 +0200
+++ /dev/fd/62 2021-05-19 20:58:22.396729637 +0200
@@ -1,12 +1,13 @@
 [Unit]
 Description=Mount unit for core18, revision 2066
 Before=snapd.service
+After=zfs-mount.service

 [Mount]
 What=/var/lib/snapd/snaps/core18_2066.snap
 Where=/snap/core18/2066
 Type=squashfs
-Options=nodev,ro,x-gdu.hide
+Options=nodev,ro,x-gdu.hide,x-gvfs-hide
 LazyUnmount=yes

 [Install]
support@qbs01:~$ diff -u <(snap info core core18 --abs-time) <(ssh qbs02 snap info core core18 --abs-time)
--- /dev/fd/63 2021-05-19 21:02:59.508319436 +0200
+++ /dev/fd/62 2021-05-19 21:02:59.508319436 +0200
@@ -9,7 +9,7 @@
 type: core
 snap-id: 99T7MUlRhtI3U0QFgl5mXXESAiSwt776
 tracking: latest/stable
-refresh-date: 2021-05-18T10:34:36+02:00
+refresh-date: 2021-05-16T11:45:44+02:00
 channels:
   latest/stable: 16-2.50 2021-05-13T01:10:05Z (11081) 103MB -
   latest/candidate: 16-2.50 2021-04-30T18:22:37Z (11081) 103MB -
@@ -27,7 +27,7 @@
 type: base
 snap-id: CSO04Jhav2yK0uz97cr0ipQRyqg0qQL6
 tracking: latest/stable
-refresh-date: 2021-05-18T10:34:24+02:00
+refresh-date: 2021-05-19T09:25:28+02:00
 channels:
   latest/stable: 20210507 2021-05-19T06:10:06Z (2066) 58MB -
   latest/candidate: 20210507 2021-05-14T09:22:35Z (2066) 58MB -

[Maciej Borzecki (maciek-borzecki)]

Most likely the mount units on qbs01 and qbs02 were generated by different versions of snapd. I suspect that qbs01 was seeded (maybe even pre-seeded) by an older version of snapd, which is why it does not have the dependency on zfs. TBH, looks like qbs02 is using an edge version of snapd?

Can you attach a similar diff for the snapd snap? Also the output of snap changes from both hosts would be useful.

[Aleksandar Ivanisevic (aleksandar-e)]

Both machines are on the same versions of everything:

support@qbs01:~$ snap list
Name Version Rev Tracking Publisher Notes
core 16-2.50 11081 latest/stable canonical✓ core
core18 20210507 2066 latest/stable canonical✓ base
lxd 4.14 20400 latest/stable canonical✓ -
support@qbs01:~$ ssh qbs02 snap list
Name Version Rev Tracking Publisher Notes
core 16-2.50 11081 latest/stable canonical* core
core18 20210507 2066 latest/stable canonical* base
lxd 4.14 20400 latest/stable canonical* -

I've been tracking the differences for a long time now, those files got different only some time after Sunday 2021-05-16, snapd itself was installed at least 6 months prior

There are no differences in "snap info snapd" output between hosts

here is the changes output

support@qbs01:~$ sudo snap changes --abs-time
ID   Status  Spawn                      Ready                      Summary
130  Done    2021-05-20T07:15:01+02:00  2021-05-20T07:15:01+02:00  Change configuration of "core" snap

support@qbs01:~$ ssh qbs02 sudo snap changes --abs-time
ID   Status  Spawn                      Ready                      Summary
140  Done    2021-05-19T09:25:19+02:00  2021-05-19T09:25:29+02:00  Refresh "core18" snap
141  Done    2021-05-20T07:15:01+02:00  2021-05-20T07:15:01+02:00  Change configuration of "core" snap

[James Henstridge (jamesh)]

This bug was filed based on discussion from the following forum thread:

    https://forum.snapcraft.io/t/core18-retagged-version/24481

So the question is whether these systemd units should be rewritten when snapd is upgraded. If we are changing the content of those unit files, then the answer should probably be yes.

[Launchpad Janitor (janitor)]

[Expired for snapd because there has been no activity for 60 days.]

[Paweł Stołowski (stolowski)]

I think the comment from James is spot on.

We don't currently regenerate all the existing mount units of installed snaps on snapd upgrade; maybe we should. This wasn't a problem so far only because our mountunit template almost never changes (the addition of After=zfs-mount.service is recent but previous changes to the generated template are 2/5/6 years old). But this is problematic.