snaps break after upgrade, snapd claims meta/snap.yaml is missing

[Alberto Donato]

On a freshly-installed 20.10 (beta) system, I'm seeing issues with snaps showing a "broken" when snapd tries to update them.

I saw this first with the LXD snap, but looking at logs it's happening with other snaps as well.

journalctl --no-pager -u snapd: https://paste.ubuntu.com/p/zGRbs9MXwg/
journalctl --no-pager -u snap.lxd.daemon: https://paste.ubuntu.com/p/YVKFMdnKg8/

$ snap version
snap 2.47.1+20.10.1
snapd 2.47.1+20.10.1
series 16
ubuntu 20.10
kernel 5.8.0-25-generic

$ snap tasks 46
Status Spawn Ready Summary
Done today at 09:12 CEST today at 09:13 CEST Ensure prerequisites for "lxd" are available
Undone today at 09:12 CEST today at 09:13 CEST Prepare snap "" (17936)
Undone today at 09:12 CEST today at 09:13 CEST Run pre-refresh hook of "lxd" snap if present
Undone today at 09:12 CEST today at 09:13 CEST Stop snap "lxd" services
Undone today at 09:12 CEST today at 09:13 CEST Remove aliases for snap "lxd"
Undone today at 09:12 CEST today at 09:13 CEST Make current revision for snap "lxd" unavailable
Undone today at 09:12 CEST today at 09:13 CEST Copy snap "lxd" data
Error today at 09:12 CEST today at 09:13 CEST Setup snap "lxd" (17936) security profiles
Hold today at 09:12 CEST today at 09:13 CEST Make snap "lxd" (17936) available to the system
Hold today at 09:12 CEST today at 09:13 CEST Automatically connect eligible plugs and slots of snap "lxd"
Hold today at 09:12 CEST today at 09:13 CEST Set automatic aliases for snap "lxd"
Hold today at 09:12 CEST today at 09:13 CEST Setup snap "lxd" aliases
Hold today at 09:12 CEST today at 09:13 CEST Run post-refresh hook of "lxd" snap if present
Hold today at 09:12 CEST today at 09:13 CEST Start snap "lxd" (17936) services
Hold today at 09:12 CEST today at 09:13 CEST Clean up "lxd" (17936) install
Hold today at 09:12 CEST today at 09:13 CEST Run configure hook of "lxd" snap if present
Hold today at 09:12 CEST today at 09:13 CEST Run health check of "lxd" snap
Done today at 09:12 CEST today at 09:13 CEST Consider re-refresh of "lxd"

......................................................................
Setup snap "lxd" (17936) security profiles

2020-10-23T09:13:06+02:00 ERROR cannot find installed snap "lxd" at revision 17936: missing file /snap/lxd/17936/meta/snap.yaml

I currently have these 2 snaps in a weird state:

$ snap list --all h2static
Name Version Rev Tracking Publisher Notes
h2static 2.2.1+git14.0a9b9b5 275 latest/edge ack disabled
h2static 279 latest/edge ack broken

$ snap list --all lxd
Name Version Rev Tracking Publisher Notes
lxd 4.7 17886 latest/stable canonical✓ -
lxd 17936 latest/stable canonical✓ disabled,broken

Note that "h2static" still seems to work correctly, but the newer snap is not mounted:

$ mount | grep h2static
/var/lib/snapd/snaps/h2static_275.snap on /snap/h2static/275 type squashfs (ro,nodev,relatime,x-gdu.hide)
nsfs on /run/snapd/ns/h2static.mnt type nsfs (rw)

and the file is not there:

$ ll /var/lib/snapd/snaps/h2static*
-rw------- 2 root root 2.5M Oct 18 18:16 /var/lib/snapd/snaps/h2static_275.snap

I've seen the same behavior with the lxd snap, and I manually reverted it to the older revision.

[Alberto Donato (ack)]

After a new stable snap revision was released for lxd, it automatically updated:

snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: yesterday at 11:00 CET
channels:
  latest/stable: 4.7 2020-10-26 (18013) 72MB -
  latest/candidate: 4.7 2020-10-26 (18013) 72MB -
  latest/beta: ↑
  latest/edge: git-740526c 2020-10-28 (18034) 72MB -
...
installed: 4.7 (18013) 72MB -

[Paweł Stołowski (stolowski)]

As far as I can tell from the logs, the problem actually affected h2static and lxd snap, and no other snaps, right?

Do you see any snap changes about h2static snap that failed? If so, can you show its tasks as well ('snap tasks ...')?

Did you try to remove (with 'snap remove') any of the affected snaps before experiencing this issue? There was a problem with remove hook of lxd snap fixed 2-3 weeks ago which - combined with https://bugs.launchpad.net/snapd/+bug/1899614 - could leave if in a broken state after failed snap remove.

[Alberto Donato (ack)]

Correct, it happened with "lxd" and "h2static".

I actually noticed it about lxd, as I couldn't use the "lxc" command (command not found).

h2static I actually noticed from "snap list" as it was showing as broken, but the command was actually working.

I haven't tried removing those snaps before they broke, this happened probably ~2 days after I reinstalled my laptop with groovy (back then, it was still beta).

for h2static reverting to the previous revision and then refreshing fixed it.

For lxd, as mentioned above, I couldn't update to 17936, but 17886 was working fine.
After 18013 was released, the snap automatically refreshed and it's since working fine.

[Alberto Donato (ack)]

I'm seeing more cases of this

$ snap list --all | grep broken
go 6715 latest/stable mwhudson disabled,broken
snapcraft 5756 latest/stable canonical* disabled,broken

A manual refresh seems to have fixed things in this case:

$ snap refresh snapcraft
snapcraft 4.4.4 from Canonical✓ refreshed
$ sudo snap refresh go
go 1.15.6 from Michael Hudson-Doyle (mwhudson) refreshed

Also https://bugs.launchpad.net/snapd/+bug/1906793 seems to point to a similar issue

[Paweł Stołowski (stolowski)]

@ack Thanks. Do you still have `snap changes` and can get `snap change <id>` for either of these snaps? It would be super useful to grab this information and logs soon after you noticed a snap that was once fine and then got broken.

[Alberto Donato (ack)]

Unfortunately I don't have those anymore. If I see snaps that get into this situation again, I'll attach them to the bug

[Paweł Stołowski (stolowski)]

Could you please attach your /var/lib/snapd/state.json next time you see this happening (you may want to remove your store macaroon from it as it's potentially sensitive data)?