do-release-upgrade from bionic->any disables lxd without snapstore access

[Dan Streetman]

[impact]

a bionic system with lxd installed, and configured, when upgraded to cosmic using do-release-upgrade, will attempt to convert the lxd deb into a snap. However, if the snapstore isn't reachable (which is a common situation in enterprise environments, where they use local apt mirrors and firewall internet access, or even are completely air-gapped), the upgrade complains loudly about being unable to access the snapstore, and the only option presented to allow install to continue is to skip or abort lxd upgrade, which states lxd will be unusable after the upgrade.

[test case]

see bug 1831933 for testbed setup (use a libvirt vm with nwfilter blocking api.snapcraft.io)

-create VM using the bionic cloud image
-in the bionic guest:
$ sudo apt update
$ sudo apt dist-upgrade

-note: snapd and lxd should already be installed; verify this
$ dpkg -l |grep -E '^ii (lxd|snapd)'
ii lxd 3.0.3-0ubuntu1~18.04.1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 3.0.3-0ubuntu1~18.04.1 amd64 Container hypervisor based on LXC - client
ii snapd 2.39.2+18.04 amd64 Daemon and tooling that enable snap packages

-edit /etc/update-manager/release-upgrades to set Prompt=normal
-reboot, if needed due to kernel upgrade
-upgrade:
$ sudo do-release-upgrade

Once the upgrade reaches the lxd package, it will display:

--
Your system is unable to reach the snap store, please make sure you're
connected to the Internet and update any firewall or proxy settings as
needed so that you can reach the snap store.

You can manually check for connectivity by running "snap info lxd"

Aborting will cause the upgrade to fail and will require it to be re-
attempted once snapd is functional on the system.

Skipping will let the package upgrade continue but the LXD commands will
not be functional until the LXD snap is installed. Skipping is allowed
only when LXD is not activated on the system.

Unable to reach the snap store
--

With the possible choices of:
Retry
Abort
Skip

Retry, of course, will never succeed.

Skip is not allowed if lxd has been configured (i.e. 'lxd init' was run, and /var/lib/lxd is populated). If lxd (and/or lxd-client) are installed but not configured, skip can be chosen to continue the upgrade, and after upgrade is complete, the lxd package will not be installed. If skip is attempted when lxd has been initialized, this is shown:

--
Skipping is not allowed when LXD has been initialized

LXD appears to have been configured on this system. Please stop LXD and
remove local data in /var/lib/lxd/ if you would like to skip installing
the LXD snap and migrating the local data.
--

Abort continues the upgrade remaining packages, after asking if the lxd upgrade failure report should be sent. However, at the end of the remaining package upgrades, the upgrade reports error and exits; it is unclear if the system is in a good fully-upgraded state at this point (besides not having an upgraded lxd, of course).

[regression potential]

TBD

[other info]

this is related to bug 1831933, but much more serious, if lxd really is unusable after the upgrade.

I targeted this bug as affecting cosmic, disco, and eoan, however currently the upgrade from bionic must go through cosmic first. And since the lxd upgrade from b->c fails without snapstore access, and there is no way to install lxd as a deb into cosmic, this bug really is just about upgrading *from* bionic.

This bug is unlikely to be seen by any enterprise customers right now, as most of them stick to LTS releases. However in 2020 once the LTS series 20.04 is released, some enterprise customers whose deployment is firewalled and/or airgapped may start to see this during upgrade.

[Dan Streetman (ddstreet)]

I suspect for lxd, this is likely a WontFix; for cosmic and later, it's provided via snap, and unlikely to go back to a deb.

I targeted this against u-r-u as well, as the 'best' fix for this may be to add a check at the start of the upgrade process, and let the upgrader choose at that point what to do, instead of waiting until mid-upgrade to let them know they need to abort the upgrade.

Additionally, whatever warning/error message is printed, either by lxd package->snap upgrade itself, and/or do-release-upgrade, should mention the snap proxy, as enterprise customers with a firewall setup will likely need to use the snap proxy.
https://docs.ubuntu.com/snap-store-proxy/en/

[Dan Streetman (ddstreet)]

> and let the upgrader choose at that point what to do

by 'upgrader' i meant the actual person doing the upgrade should be notified, and choose what to do, before upgrade starts

[Stéphane Graber (stgraber)]

Hi,

Indeed, this is a won't fix for LXD as we are snap only at this point and our preinst behavior is I believe correct in avoiding breaking systems. The post-upgrade deb is completely empty so allowing the update to proceed will completely break the existing LXD. The only options are then effectively to Abort so LXD fails in preinst and your running instance keeps working or get your firewall/proxy sorted and hit Retry.

Skip was added somewhat recently to handle WSL I believe and indeed allows skipping the snap installation if the system doesn't currently use LXD.

I agree that the release upgrader should perform those checks ahead of time if it knows that some packages will have to be migrated to snaps and not allow the user to keep doing the upgrade without having sorted out store access or removed the affected packages.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Ehrhardt  (paelzer)]

This breaks me quite often as well.
Even in cases where I "COULD" set up a httpproxy.
Let me add a showcase:
1. Bionic was fine, apt and software installs worked fine
   E.g. the network was setup to allow the archive.ubuntu.com in the FW
   But not for anything else (e.g. snapstore)
2. do-release-upgrade will fail half way through with all the pain of broken installs and such. Really painful to recover and a bad experience

I was wondering if Bionic's do-release-upgrade should check and only get started if it can verify that snapd can reach the store (or enterprise proxy or any else).
This won't solve all of the original bug (air-gapped still have issues) but it would avoid a lot of people that are not air-gapped but have special handling for archive.ubuntu.com in place yet nothing for the snapstore to fight this in a broken upgrade. They could adapt the setup as needed upfront and then have a smooth experience on the upgrade.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.12

---------------
ubuntu-release-upgrader (1:20.04.12) focal; urgency=medium

  * data/release-upgrades: set Prompt to lts as focal is an LTS release.
  * DistUpgrade/DistUpgradeQuirks.py: Do not allow the upgrade to proceed if
    the non-transitional lxd deb is installed and we can not acces the Snap
    Store. (LP: #1834501)

 -- Brian Murray <email address hidden> Mon, 17 Feb 2020 14:11:05 -0800

[Brian Murray (brian-murray)]

I tested an upgrade from 18.04 to 19.10 with lxd installed and access to api.snapcraft.io blocked. When presented with the lxd installation question I chose to abort, the upgrade completed and seemed fine. I then reenabled access to api.snapcraft.io and ran 'dpkg --configure -a' (or something close to that) and lxd transitioned to the snap. So given that it is possible to recover from this situation and that not many people will be upgrading from 18.04 to 19.10 we are going to set this to Won't Fix. That being said it'd be trivial for somebody else to backport my fix in 20.04 to 19.10.

[Patrick (patrick123)]

I just encountered this, attempting to upgrade 18.04 to 20.04. For me, the problem is that I have an IPv6-only system, but api.snapcraft.io is not reachable via IPv6.