ubuntu-release-upgrader failed to upgrade snap to track new release

[Erich Eickmeyer]

When upgrading Edubuntu 23.04 to Edubuntu 23.10 via `update-manager -d`, it failed to stop the snap-store and refresh it to track latest/stable/ubuntu-23.10 and was left still tracking latest/stable/ubuntu-23.04.

This is problematic as, if ubuntu-release-upgrader cannot stop snaps to upgrade them during the upgrade process, then snaps will be stuck tracking old versions.

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: ubuntu-release-upgrader-core 1:23.10.6
ProcVersionSignature: Ubuntu 6.5.0-5.5-generic 6.5.0
Uname: Linux 6.5.0-5-generic x86_64
ApportVersion: 2.27.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: pass
CrashDB: ubuntu
CrashReports:
 640:0:0:12097:2023-09-16 09:29:18.827962636 -0700:2023-09-16 09:29:19.827962636 -0700:/var/crash/tuxedo-keyboard.0.crash
 600:0:0:273896:2023-09-16 09:28:56.223826885 -0700:2023-09-16 09:28:57.223826885 -0700:/var/crash/linux-headers-6.5.0-5-generic.0.crash
CurrentDesktop: ubuntu:GNOME
Date: Wed Sep 20 08:23:02 2023
InstallationDate: Installed on 2023-03-28 (177 days ago)
InstallationMedia: Edubuntu 23.04 "Lunar Lobster" - Alpha amd64 Binary-1 (20230327)
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: Upgraded to mantic on 2023-09-09 (11 days ago)
VarLogDistupgradeTermlog:

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeremy Bícha (jbicha)]

I confirm this with Ubuntu Desktop 23.04 to 23.10

[Brian Murray (brian-murray)]

I found something valuable in my terminal when running do-release-upgrade:

refreshing snap snap-store
error: cannot perform the following tasks:
- Setup snap "snap-store" (1043) security profiles (cannot find installed snap "snap-store" at revision 959: missing file /snap/snap-store/959/meta/snap.yaml)
- Automatically connect eligible plugs and slots of snap "snap-store" (internal error: auto-connect of &{"snap-store:desktop-launch" "snapd:desktop-launch"} failed: snap "snapd" has no slot named "desktop-launch")

So I wonder if this is really an issue with ubuntu-release-upgrader.

[Brian Murray (brian-murray)]

There is also an when refreshing firefox:

refreshing snap firefox
error: cannot perform the following tasks:
- Setup snap "firefox" (3145) security profiles (cannot find installed snap "firefox" at revision 2517: missing file /snap/firefox/2517/meta/snap.yaml)
- Run post-refresh hook of "firefox" snap if present (run hook "post-refresh": cannot perform operation: mount --rbind /dev /tmp/snap.rootfs_TbK5Bd//dev: No such file or directory)

And the firefox snap is still tracking a 23.04 channel:

bdmurray@clean-lunar-amd64:/var/log/dist-upgrade$ snap info firefox
name: firefox
summary: ""
publisher: Mozilla\u2713
store-url: https://snapcraft.io/firefox
license: unset
description: |

snap-id: 3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk
tracking: latest/stable/ubuntu-23.04
refresh-date: today at 12:24 PDT
channels:
  latest/stable: 117.0.1-2 2023-09-12 (3131) 248MB -
  latest/candidate: 118.0-1 2023-09-18 (3166) 251MB -
  latest/beta: 118.0b9-1 2023-09-15 (3150) 251MB -
  latest/edge: 119.0a1 2023-09-20 (3176) 269MB -
  esr/stable: 115.2.1esr-1 2023-09-12 (3130) 253MB -
  esr/candidate: 115.3.0esr-1 2023-09-18 (3163) 253MB -
  esr/beta: \u2191
  esr/edge: \u2191
installed: (2517) 0B broken

[Nick Rosbrook (enr0n)]

Looking at the code in u-r-u, if e.g. firefox is tracking stable/ubuntu-23.04, then u-r-u will simply try:

snap refresh --channel stable/ubuntu-23.10 firefox

This seems like the right thing to do, but maybe we're missing a subtlety.

[Erich Eickmeyer (eeickmeyer)]

While that might be the right thing to do, snap will simply balk if e.g. firefox is open, and will certainly balk since snap-store runs in the background. It would have to force-close those in order to refresh those.

[Sergio Schvezov (sergiusens)]

Not an authority, but perhaps doing a "snap switch" can help here (as it does not perform the refresh), I suspect there is a way to disable refresh awareness.

[Michael Vogt (mvo)]

I see two options:
a) just use "--ignore-running", i.e. "snap refresh --channel=... --ignore-running firefox"
b) use "snap switch --channel=..." and just let snapd deal with it on the next auto-refresh

Both have pros/cons, but I would lean slightly towards (a) because the release upgrader is already telling users to close apps (iirc?) and that running apps may become unstable. In general I would also assume that stable and stable/ubuntu-23.10 point to the same revision so for the most part it would be a no-op. And when it's not there is probably a good reason so we should upgrade.

Hope this helps!

[Samuele Pedroni (pedronis)]

> refreshing snap snap-store
error: cannot perform the following tasks:
- Setup snap "snap-store" (1043) security profiles (cannot find installed snap "snap-store" at revision 959: missing file /snap/snap-store/959/meta/snap.yaml)
- Automatically connect eligible plugs and slots of snap "snap-store" (internal error: auto-connect of &{"snap-store:desktop-launch" "snapd:desktop-launch"} failed: snap "snapd" has no slot named "desktop-launch")

the first part of that error means the snap is not mounted properly, which indicates a more serious problem in terms of getting the snaps in a bad state

We would need system logs and snapd logs with SNAPD_DEBUG=1 set of the snapd running of a system during the upgrade that fails

[Michael Vogt (mvo)]

One thing worth checking in the "main.log" is that it ends with:
"""
2023-09-09 11:15:25,169 DEBUG refresh of snap snapd-desktop-integration succeeded
"""
so it seems like something exists prematurely. Looking at the code it seems to happen inside/after
"_replaceDebsAndSnaps(self)", given that there is no exception it's a bit odd that it exists there, after the quirks there should be the cleanup stage which should produce logs.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nick Rosbrook (enr0n)]

Samuele - I have attached the output of `journalctl -u snapd.service -b --since <time upgrade started>`. Please let me know what other logs you would like.

[Nick Rosbrook (enr0n)]

For the short term, I think that doing `snap switch` instead of `snap refresh` is the way to go. An experiment with `--ignore-running` had the same issue.

[Erich Eickmeyer (eeickmeyer)]

`snap switch` might actually be the right way to go. The next time `snap refresh` runs, which would be the next time `unattended-upgrades` runs, would be when these snaps would get upgraded (in theory). I like this idea.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2036765

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:23.10.7

---------------
ubuntu-release-upgrader (1:23.10.7) mantic; urgency=medium

  * DistUpgradeQuirks: Use generic font temporarily at upgrade
    (LP: #2034986)
  * DistUpgradeQuirks: Switch snap channels instead of refresh
    (LP: #2036765)
  * DistUpgradeController: Ensure security archive is used for security pocket
    (LP: #2036679)
  * Run pre-build.sh: updating mirrors and demotions.

 -- Nick Rosbrook <email address hidden> Fri, 29 Sep 2023 14:44:34 -0400

[Nedim (nedeem)]

This bug was affecting me:

````
nedim@T580:~$ snap switch --channel=latest/stable/ubuntu-23.10 snap-store
````

I would get this error

````
error: snap "snap-store" is not installed
````

Solved by

````
snap install snap-store
snap switch --channel=latest/stable/ubuntu-23.10 snap-store

````

[Francis De Brabandere (francisdb)]

I wonder what other app updates I missed during upgrade?

For the firmware updater

```
sudo snap install firmware-updater
```

[Boris Erdmann (boris-erdmann)]

The bug is still present. Upgraded on 2023-10-13 so I am stuck on old channels now for all these

$ snap list --all
Name Version Revision Tracking Herausgeber Hinweise
bare 1.0 5 latest/stable canonical✓ base
chromium 118.0.5993.70 2666 latest/stable canonical✓ -
core 16-2.60.4 16202 latest/stable canonical✓ core
core18 20230703 2790 latest/stable canonical✓ base
core20 20230801 2015 latest/stable canonical✓ base
core22 20230801 864 latest/stable canonical✓ base
cups 2.4.6-4 980 latest/stable openprinting✓ -
firefox 118.0.2-2 3252 latest/stable mozilla✓ -
gnome-3-28-1804 3.28.0-19-g98f9e67.98f9e67 198 latest/stable canonical✓ -
gnome-3-38-2004 0+git.efb213a 143 latest/stable canonical✓ -
gnome-42-2204 0+git.ff35a85 141 latest/stable canonical✓ -
gtk-common-themes 0.1-81-g442e511 1535 latest/stable canonical✓ -
hunspell-dictionaries-1-7-2004 1.7-20.04+pkg-6fd6 2 latest/stable brlin -
snap-store 41.3-71-g709398e 959 latest/stable canonical✓ -
snapd 2.60.4 20290 latest/stable canonical✓ snapd
snapd-desktop-integration 0.9 83 latest/stable canonical✓ -

or only some of them?

Do they all have to be switched to latest/stable/ubuntu-23.10 ??

[Boris Erdmann (boris-erdmann)]

Also, @Francis De Brabandere (francisdb) post-install for the firmware updater seems to install a very outdated version of the firmware updater given the screenshot here: https://snapcraft.io/install/firmware-updater/ubuntu

[Phil Hughes (phil-hughes)]

I can confirm that I had this on a PC upgraded 2023-11-12.

Can someone confirm that all snaps should be tracking channel=latest/stable/ubuntu-23.10? Thanks.

[Oliver Calder (ocalder)]

I saw the same problem occur for me on upgrade from lunar to mantic on 14 November.

In my case, snap-store remained tracking latest/stable (as opposed to latest/stable/ubuntu-23.10), but I did end up with firmware-updater successfully installed and tracking latest/stable/ubuntu-23.10.

I manually refreshed snap-store to channel=latest/stable/ubuntu-23.10, and all was well.