snaps auto-refresh during do-release-upgrade results in broken looping update

[Dimitri John Ledkov]

I was on Ubuntu Jammy desktop.
I am a normal Ubuntu user - with snapd desktop integration, mattermost snap, canonical livepatch snap with desktop integration, etc.
All of the snaps open and running.

Started do-release-upgrade to nobble.

During the upgrade snap auto-refresh kicked in, whilst do-release-upgrade was also trying to refresh and update channels.

Things did not go well, as I had stuck namespaces preventing removal of snaps - and I couldn't even tell why snaps were being attempted to be removed.

ID Status Spawn Ready Summary
283 Error yesterday at 22:11 GMT yesterday at 23:26 GMT Auto-refresh snaps "core", "firefox", "snapcraft"
290 Error yesterday at 22:17 GMT yesterday at 23:26 GMT Remove "gnome-3-38-2004" snap

Note that dist upgrade stated during auto-refresh happening

$ snap tasks 283
Status Spawn Ready Summary
Done yesterday at 22:11 GMT yesterday at 23:26 GMT Ensure prerequisites for "core" are available
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Download snap "core" (16574) from channel "latest/stable"
Done yesterday at 22:11 GMT yesterday at 23:26 GMT Fetch and check assertions for snap "core" (16574)
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Mount snap "core" (16574)
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Run pre-refresh hook of "core" snap if present
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Stop snap "core" services
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Remove aliases for snap "core"
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Make current revision for snap "core" unavailable
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Copy snap "core" data
Error yesterday at 22:11 GMT yesterday at 23:26 GMT Setup snap "core" (16574) security profiles
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Make snap "core" (16574) available to the system
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Automatically connect eligible plugs and slots of snap "core"
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Set automatic aliases for snap "core"
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Setup snap "core" aliases
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Run post-refresh hook of "core" snap if present
Undone yesterday at 22:11 GMT yesterday at 23:26 GMT Start snap "core" (16574) services
Done yesterday at 22:11 GMT yesterday at 23:26 GMT Remove data for snap "core" (16091)
Hold yesterday at 22:11 GMT yesterday at 23:26 GMT Remove snap "core" (16091) from the system
Hold yesterday at 22:11 GMT yesterday at 23:26 GMT Clean up "core" (16574) install

......................................................................
Remove snap "core" (16091) from the system

2024-02-07T22:59:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:02:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:05:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:08:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:11:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:14:29Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:17:30Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:20:30Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:23:30Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.
2024-02-07T23:26:30Z ERROR cannot remove snap file "core", will retry in 3 mins: umount: /snap/core/16091: not mounted.

$ snap tasks 290
Status Spawn Ready Summary
Undone yesterday at 22:17 GMT yesterday at 23:26 GMT Stop snap "gnome-3-38-2004" services
Undone yesterday at 22:17 GMT yesterday at 23:26 GMT Run remove hook of "gnome-3-38-2004" snap if present
Done yesterday at 22:17 GMT yesterday at 23:26 GMT Disconnect interfaces of snap "gnome-3-38-2004"
Undone yesterday at 22:17 GMT yesterday at 23:26 GMT Save data of snap "gnome-3-38-2004" in automatic snapshot set #2
Undone yesterday at 22:17 GMT yesterday at 23:26 GMT Remove aliases for snap "gnome-3-38-2004"
Undone yesterday at 22:17 GMT yesterday at 23:26 GMT Make snap "gnome-3-38-2004" unavailable to the system
Error yesterday at 22:17 GMT yesterday at 23:26 GMT Remove security profile for snap "gnome-3-38-2004" (143)
Done yesterday at 22:17 GMT yesterday at 23:26 GMT Remove data for snap "gnome-3-38-2004" (140)
Hold yesterday at 22:17 GMT yesterday at 23:26 GMT Remove snap "gnome-3-38-2004" (140) from the system
Hold yesterday at 22:17 GMT yesterday at 23:26 GMT Remove data for snap "gnome-3-38-2004" (143)
Hold yesterday at 22:17 GMT yesterday at 23:26 GMT Remove snap "gnome-3-38-2004" (143) from the system

......................................................................
Remove security profile for snap "gnome-3-38-2004" (143)

2024-02-07T23:26:17Z ERROR cannot find installed snap "gnome-3-38-2004" at revision 143: missing file /snap/gnome-3-38-2004/143/meta/snap.yaml

......................................................................
Remove snap "gnome-3-38-2004" (140) from the system

2024-02-07T22:59:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:02:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:05:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:08:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:11:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:14:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:17:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:20:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:23:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.
2024-02-07T23:26:04Z ERROR cannot remove snap file "gnome-3-38-2004", will retry in 3 mins: umount: /snap/gnome-3-38-2004/140: not mounted.

Trying to get things going by unmounting things, changing namespaces did not work.

I did not try /usr/lib/snapd/snap-discard-ns during that, as I didn't know it was a thing.

I did kill `snap remove gnome-3-38-2004` to get the do-release-upgrade to keep going, as I wanted that to complete.

Because otherwise everything was stuck on the 3min loops of unable to remove snaps forever (or at least so it seemed).

Overall it was a bad experience. I don't know if any of the above things are bugs in snapd. But this bug is that likely do-release-upgrade needs better snap integration: either trigger auto-refresh of all snaps (like it requires to apply all .deb updates), or it should hold auto-refresh, or maybe both refresh everything followed by holding auto-refresh.

Given do-release-upgrade changes tracks of things, and there might be migration hooks in snaps, imho it seems better to ensure that all snaps are updated prior to do-release-upgrade. Which was not the case for me.