Unable to contact snap store on Xubuntu from Jammy to Noble

[Skia]

[Impact]

Upgrades from Jammy to Noble can fail if snap transition packages are installed after the new systemd has been unpacked, but before systemd-resolved has been unpacked.

In practice, this manifests on at least Xubuntu upgrades during the chromium-browser snap transition.

[Test Plan]

Without this fix, the snap store connectivity check will fail when installing the chromium-browser snap because systemd-resolved is not present, so DNS lookups fail during the connectivity check.

With the fix in place, the systemd-resolved stub resolver should be disabled during the upgrade so that DNS lookups succeed even when systemd-resolved is not yet unpacked on the system again.

So, to test this fix, we should run an upgrade on a Xubuntu 22.04 system:

$ do-release-upgrade -d

And verify that (a) the chromium-browser installation (and snap store connectivity check) succeeds, and (b) that the stub resolver is not in use during the upgrade.

$ cat /etc/resolv.conf
$ cat /run/systemd/resolved.conf.d/dist-upgrade-no-stub-resolver.conf

[Where problems could occur]

This patch disables the systemd-resolved stub resolver by writing a drop-in to
/run/systemd/resolved.conf.d. If there was a mistake in the drop-in, it would not correctly disable the stub resolver.

[Original Description]

When running `do-release-upgrade -d` on Jammy Xubuntu, the upgrade fails consistently at the `Installing the firefox snap` step.

This was first detected on Jenkins [1], then reproduced locally with `auto-upgrade-testing`, then finally by manually running a `do-release-upgrade -d` in a VM.

[1]: https://platform-qa-jenkins.ps5.ubuntu.com/view/upgrade-list/job/upgrade_ubuntu-jammy-noble-xubuntu-amd64_qemu/7/consoleFull (Jenkins full log attached)

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: ubuntu-release-upgrader-core 1:22.04.17
ProcVersionSignature: User Name 5.15.0-88.98-generic 5.15.126
Uname: Linux 5.15.0-88-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
CrashDB: ubuntu
CurrentDesktop: XFCE
Date: Fri Nov 17 18:24:22 2023
PackageArchitecture: all
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Murray (brian-murray)]

Looking at the log files this is happening when installing the firefox deb so the problem is likely there.

06:31:33 Selecting previously unselected package firefox.
06:31:33 Preparing to unpack .../26-firefox_1%3a1snap1-0ubuntu5_amd64.deb ...
06:31:33 => Installing the firefox snap
06:31:33 ==> Checking connectivity with the snap store
06:32:06 ===> Unable to contact the store, trying every minute for the next 30 minutes
06:47:28 ===> Still unable to contact the store, trying for another 20 minutes
07:02:51 ===> Still unable to contact the store, trying for another 10 minutes
07:16:00 Build timed out (after 60 minutes). Marking the build as failed.

[Julian Andres Klode (juliank)]

So the script here fails to run

snap info firefox

It was pointed out it managed to upgrade the chromium snap before it, so not sure what's going on, this seems like a job for the snapd team to investigate.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Erich Eickmeyer (eeickmeyer)]

This happened during a test upgrade from Ubuntu Studio 22.04 -> 24.04, but instead of Firefox, it tried to pull-in chromium??? We don't even seed chromium in any way.

[Brian Murray (brian-murray)]

Erich - Do you have /var/log/dist-upgrade/main.log for the system which was being upgraded? That would help us evaluate the failure.

[Skia (hyask)]

Here is the whole Jenkins archive. The file you asked is in `archive/ubuntu-jammy-noble-xubuntu-amd64_qemu/20240105.141607/artifacts/upgrade_run/system_details/dist-upgrade/main.log`, along with many other files that may also be interesting.

[Tim Andersson (andersson123)]

Tried this again today, still an issue, see files attached

[Gabriel Korytiak (gabriel-korytiak)]

have anyone tried to remove firefox snap and deb package before upgrade ?

sudo snap remove firefox
sudo apt purge firefox firefox-*

[Captain (captainmorgan6666)]

had the same issue I am upgrading now with after "sudo snap remove firefox" and will let you know. I did not remove the .deb firefox version.

[Skia (hyask)]

Now that upgrades from Jammy to Noble are re-enabled, the issue shows again on multiple flavors. I'm attaching the Jenkins artifacts for each jobs. Sometimes, it's when installing chromium, and sometimes firefox, but that doesn't change that the upgrade fails in the end.

[Gabriel Korytiak (gabriel-korytiak)]

you must remove both deb packages and snaps otherwise it will cause issues

before upgrade do this :

sudo apt purge firefox firefox-*
sudo snap remove firefox

upgrade to 24.04
after upgrade add this mozilla respository

https://launchpad.net/~mozillateam/+archive/ubuntu/ppa

sudo apt install firefox firefox-locale-en

this should bypass snap instalation and allow you to upgrade

[Skia (hyask)]

Hi Gabriel,

Thanks for your proposed solution. While this may be fine for end users to do so, let me highlight some things people should know if they read this:
* This is a manual solution that could be applied to a system, but please take good care in checking that removing `firefox` doesn't remove your whole desktop, since some metapackages like `ubuntu-desktop` seems to depend on it. More generally, this solution is more a hack that might work for some people, but you better know what you're doing.
* Adding the Mozilla third-party repository may also be fine, but is not officially supported. This will install Firefox as a native package, while the official supported way on Ubuntu is with the snap. That mostly means that this is way less tested, and you might have other unknown issues there. You also need to evaluate the trust you put into that third party repo.
* This bug was opened because our automated testing is broken, so the root-cause needs to be fixed anyway. This is not really an end-user support thread.

[Gabriel Korytiak (gabriel-korytiak)]

ok fine i maybe missunderstaned something here. I trought firefox was causing instalation to broke or fail and thats why i recomended to remove temporarely firefox and reinstalled it back after OS upgrade.

Ubuntu-desktop dont depend on firefox and i can remove i freely without any issues or breakage of GUI or entire OS

forcing snaps isnt a good thing and never will be

what i recommended is still only a temporary bypass and it will indeed need a proper fix

[Skia (hyask)]

This is still reproducible easily on the following flavors:
  * kubuntu
  * lubuntu
  * mate
  * xubuntu

[Andrew Phelps (andrewphelpsj)]

Hello, I was able to reproduce this issue when upgrading MATE from Jammy to Noble.

By manually running "snap info firefox" during the upgrade process, I could see it failing:

$ snap info firefox
error: no snap found for "firefox"

This error message isn't great, but it is what we report right now from "snap info" when we're unable to reach the store.

This issue wasn't unique to snapd though, I wasn't able to resolve any DNS queries at all.

$ nslookup snapcraft.io
communications error to 127.0.0.53#53: connection refused
communications error to 127.0.0.53#53: connection refused
communications error to 127.0.0.53#53: connection refused
no servers could be reached

It looks like systemd-resolved was in some sort of broken state, here are its logs:

systemd-resolved.service: Current command vanished from the unit file, execution of the command list won't be resumed.
systemd-resolved.service: Got notification message from PID 344, but reception is disabled.
systemd-resolved.service: Got notification message from PID 344, but reception is disabled.
systemd-resolved.service: Got notification message from PID 344, but reception is disabled.
systemd-resolved.service: Watchdog timeout (limit 3min)!
systemd-resolved.service: Killing process 344 (systemd-resolve) with signal SIGABRT.
systemd-resolved.service: Main process exited, code=dumped, status=6/ABRT
systemd-resolved.service: Failed with result 'watchdog'.

/etc/resolv.conf was still the systemd-resolved generated one. When I replaced it with "nameserver 8.8.8.8", then "snap info firefox" succeeded and I was able to continue with the release upgrade.

[Nick Rosbrook (enr0n)]

I just tried an upgrade on Xubuntu from Jammy to Noble. The reason this happens is that in Kinetic, systemd-resolved was split into its own binary package, and at the time that the transitional deb(s) are being installed for thunderbird or firefox etc. (and hence the snap store connectivity check happens), systemd-resolved has not been installed yet. In other words, we have the following order of events:

1. New systemd unpacked / installed (results in systemd-resolved.service being killed and removed)
2. Snap transitional packages installed, snap store connectivity check attempted
3. New systemd-resolved unpacked / installed (systemd-resolved.service restored)

But we need systemd-resolved to be back before the connectivity check (or have some other workaround in place).

[Nick Rosbrook (enr0n)]

ProblemType: Bug
ApportVersion: 2.28.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: pass
CrashDB: ubuntu
CurrentDesktop: XFCE
DistroRelease: Ubuntu 24.04
InstallationDate: Installed on 2024-07-30 (0 days ago)
InstallationMedia: Xubuntu 22.04.4 LTS "Jammy Jellyfish" - Release amd64 (20240216.1)
Package: ubuntu-release-upgrader (not installed)
ProcVersionSignature: Ubuntu 6.5.0-45.45~22.04.1-generic 6.5.13
RebootRequiredPkgs: Error: path contained symlinks.
Tags: noble dist-upgrade
Uname: Linux 6.5.0-45-generic x86_64
UpgradeStatus: Upgraded to noble on 2024-07-30 (0 days ago)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
VarLogDistupgradeXorgFixuplog:
 INFO:root:/usr/bin/do-release-upgrade running
 INFO:root:No xorg.conf, exiting
_MarkForUpload: True
modified.conffile..etc.init.d.apport: [modified]
mtime.conffile..etc.init.d.apport: 2024-04-23T07:30:10

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

apport information

[Nick Rosbrook (enr0n)]

I think a viable workaround for this, and something we can implement as a quirk in ubuntu-release-upgrader, is to disable the systemd-resolved stub resolver during the upgrade. This will leave behind a "traditional", usable /etc/resolv.conf even after systemd-resolved is gone.

That should allow the snap store connectivity check to succeed.

[Steve Langasek (vorlon)]

Did you rule out the option of early-install'in systemd-resolved during the upgrade? Then there wouldn't need to be special-casing of resolv.conf because it wouldn't be "gone"

[Nick Rosbrook (enr0n)]

> Did you rule out the option of early-install'in systemd-resolved during the upgrade?

My assessment was that this would be too complicated in practice. While there are many mechanisms in the upgrader to *mark* additional packages for installation, there is no good way to say "install this package right now" early in the upgrade. Nor is there a way to change the ordering of packages installed during the upgrade.

The advantage of disabling the stub resolver is that (a) it leaves the apt cache and upgrade calculation alone, and (b) it's a supported mode for systemd-resolved anyways. We're not doing anything tricky; just transiently setting `DNSStubListener=no`.

[Chris Halse Rogers (raof)]

An upload of ubuntu-release-upgrader to noble-proposed has been rejected from the upload queue for the following reason: "Changing the "The package %s is marked for removal..." string to an f"" string unnecessarily breaks translations, and it looks like the Amharic translation got broken (it retains the old translation, containing a '%s' - I'm not *sure* what python will do when it hits that, but it's not going to be what we want). Otherwise looks fine to me.".

[Łukasz Zemczak (sil2100)]

Hello Skia, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

I have verified the fix using ubuntu-release-upgrader from noble-proposed using a xubuntu 22.04 VM:

ubuntu@xubuntu:~$ sudo sed -i 's/Prompt=lts/Prompt=normal/g' /etc/update-manager/release-upgrades
[sudo] password for ubuntu:
ubuntu@xubuntu:~$ do-release-upgrade --proposed
Checking for a new Ubuntu release

= Welcome to Ubuntu 24.04 LTS 'Noble Numbat' =

The Ubuntu team is proud to announce Ubuntu 24.04 LTS 'Noble Numbat'.

To see what's new in this release, visit:
  https://wiki.ubuntu.com/NobleNumbat/ReleaseNotes

[ ...SNIP... ]

In another terminal, while the upgrade was running:

ubuntu@xubuntu:~$ cat /etc/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 10.19.111.1
nameserver fd42:3cda:638d:a9b3::1
nameserver fe80::216:3eff:fe07:85b6%2
search lxd
ubuntu@xubuntu:~$ cat /run/systemd/resolved.conf.d/dist-upgrade-no-stub-resolver.conf
# Temporarily disable stub resolver during upgrade
[Resolve]
DNSStubListener=no
ubuntu@xubuntu:~$ tail /var/log/dist-upgrade/main.log
2024-08-12 16:21:57,233 DEBUG dir '/tmp' needs '0.0' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x731e8275c940>' (14185896140.600000)
2024-08-12 16:21:57,233 DEBUG dir '/usr' needs '0.0' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x731e8275c940>' (14185896140.600000)
2024-08-12 16:21:57,233 DEBUG Found writable ESP /dev/sda1 /boot/efi vfat rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
2024-08-12 16:21:57,242 DEBUG quirks: running StartUpgrade
2024-08-12 16:21:57,242 DEBUG running Quirks.StartUpgrade
2024-08-12 16:21:57,243 DEBUG skipping 'README' (no '.')
2024-08-12 16:21:57,244 DEBUG killing update-notifier
2024-08-12 16:21:57,248 DEBUG killing kblueplugd kbluetooth4
2024-08-12 16:21:57,251 DEBUG Disabling systemd-resolved stub resolver during upgrade
2024-08-12 16:21:57,354 INFO cache.commit()

And after the upgrade was complete:

ubuntu@xubuntu:~$ apt policy chromium-browser
chromium-browser:
  Installed: 2:1snap1-0ubuntu2
  Candidate: 2:1snap1-0ubuntu2
  Version table:
 *** 2:1snap1-0ubuntu2 500
        500 http://us.archive.ubuntu.com/ubuntu noble/universe amd64 Packages
        100 /var/lib/dpkg/status
ubuntu@xubuntu:~$ snap info chromium
name: chromium
summary: Chromium web browser, open-source version of Chrome
publisher: Canonical✓
store-url: https://snapcraft.io/chromium
contact: https://bugs.launchp...