LXD Deb->snap transition fails in WSL due to snap command not working (yet)

[Balint Reczey]

[Impact]

 * Upgrading .deb-packaged lxd to the lxd .deb package that just installs the snap fails when the lxd snap can't be installed.

 * This breaks package upgrades and specifically do-release-upgrade in the middle of the release upgrade.

 * The fix lets skipping the installation of the lxd snap if lxd has not been activated on the system.

[Test Case]

1.
 * Install the Ubuntu 18.04 app from the Microsoft Store on Windows 10
 * Upgrade the packages to the latest versions in 18.04
 * Make sure the lxd package is installed
 * Run sudo do-release-upgrade
 * Observe the release upgrade failing without the fix and offering skipping the snap installation and then succeeding after skipping it.
 * Observe the lxd package still being installed but the lxc and lxd commands asking for installing the lxd snap
 * Pretend that there is an activated lxd server:
  # mkdir /var/lib/lxd
  # touch /var/lib/lxd/server.crt
 * Reinstall the lxd package
  # apt-get install --reinstall lxd
 * Observe the installation not allowing to skip the installation of the lxd snap. (By removing the created file skipping can be made allowed again.)

2.
 * On a fully upgraded cosmic system install the previous version of the package, then upgrade to the fixed version.
 * Observe the upgrade succeeding

3.
 * On a fully upgraded cosmic system install the fixed version of the package, then remove the lxd snap.
 * Observe the lxc and lxd commands asking for installing the lxd snap.

4.
 * On a fully upgraded cosmic system purge the lxd packge and remove the lxd snap.
* Install the fixed package and observe it installing the lxd snap as well.

[Regression Potential]

 * The fix includes changing the packaged-shipped lxc and lxd commands to print an error message if the lxd snap is missing. This is very simple and unlikely to cause regressions.
* The change to the package's prerm script is more extensive and in case of a programming error error it may cause upgrade/installation failures. The provided tests excercise the

[Original Bug Text]

Upgrading from Bionic to Cosmic (or later) is not possible in WSL (https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux) since when lxd package upgrade (with the deb -> snap transition) starts it can't be completed successfully. The package upgrade can only be aborted if the snap command fails, hence the distribution upgrade can't be completed either.

As a workaround one can do:

 $ sudo mv /usr/bin/snap /usr/bin/snap.real
 $ sudo ln -s /bin/true /usr/bin/snap

in a separate terminal, but this is not a smooth upgrade experience by any means.

The upgrade can fail the same way if the snap store is not available possibly due to strict firewalling requirements and most likely distro upgrades should work in those environments as well.

As a solution when the snap transition does not work during the package upgrade please upgrade the package to a state, where new invocations of lxd or lxc starts the snap installation later.

A prototype can be found here:
https://code.launchpad.net/~mwhudson/+git/lxd-installer/+ref/master

[Stéphane Graber (stgraber)]

I don't believe that dist-upgrading WSL is officially supported as it's an Ubuntu installation which does not run a supported Ubuntu kernel. As you point out, LXD cannot work in such an environment and so it's reasonable for its upgrade to similarly fail.

The lxd-installer proposal also isn't suitable because lxd.migrate which performs the needed data migration from the deb over to the snap requires the previous version of LXD to be functional.

This is why we do the upgrade from preinst and prevent the system from upgrading at all unless the upgrade is processed properly. Anything else would effectively result to data loss.

In the event that the store isn't available, we offer the user the choice between retrying after they've made sure it's accessible or giving up on the upgrade until such a time they can make it available (or have manually remove LXD from their system).

[Balint Reczey (rbalint)]

On Wed, Mar 27, 2019 at 7:25 PM Stéphane Graber <email address hidden> wrote:
>
> I don't believe that dist-upgrading WSL is officially supported as it's
> an Ubuntu installation which does not run a supported Ubuntu kernel. As

Please note that to work towards the possibility of offering "official
support" first we need to fix issues like this.
I'm happy to discuss this in private. WSL is not the only Ubuntu
installation that does not run on Ubuntu-provided kernel.

> you point out, LXD cannot work in such an environment and so it's
> reasonable for its upgrade to similarly fail.

No, package upgrades must not fail in dist-upgrade this way. This is
the exact opposite of great user experience.
Also I removed the lxd package to try that as a workaround but
do-release-upgrade brought it back (which may be a separate issue).

>
> The lxd-installer proposal also isn't suitable because lxd.migrate which
> performs the needed data migration from the deb over to the snap
> requires the previous version of LXD to be functional.
>
> This is why we do the upgrade from preinst and prevent the system from
> upgrading at all unless the upgrade is processed properly. Anything else
> would effectively result to data loss.

It can be easily tested in preinst if there was data to migrate, and
if there is none the proposed switch to lxd-installer mode would just
be enough.

>
> In the event that the store isn't available, we offer the user the
> choice between retrying after they've made sure it's accessible or
> giving up on the upgrade until such a time they can make it available
> (or have manually remove LXD from their system).

This is not enough if the store access can't be fixed, for example
when the company policy prohibits opening this firewall port.
Sticking to not falling back to something and aborting the
dist-upgrade would result a lot of support queries about broken
upgrades to 20.04.

Please consider handling the case when there is not prior lxd data to
migrate gracefully and reopen the bug.

>
> ** Changed in: lxd (Ubuntu)
> Status: New => Won't Fix
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1821924
>
> Title:
> LXD Deb->snap transition fails in WSL due to snap command not working
> (yet)
>
> Status in lxd package in Ubuntu:
> Won't Fix
>
> Bug description:
> Upgrading from Bionic to Cosmic (or later) is not possible in WSL
> (https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux) fails
> since when lxd package upgrade (with the deb -> snap transition)
> starts it can't be completed successfully. The package upgrade can
> only be aborted if the snap command fails, hence the distribution
> upgrade can't be completed either.
>
> As a workaround one can do:
>
> $ sudo mv /usr/bin/snap /usr/bin/snap.real
> $ sudo ln -s /bin/true /usr/bin/snap
>
> in a separate terminal, but this is not a smooth upgrade experience by
> any means.
>
> The upgrade can fail the same way if the snap store is not available
> possibly due to strict firewalling requirements and most likely distro
> upgrades should work in those environments as...

[Stéphane Graber (stgraber)]

Re-opened as low as in the case where LXD is unused, we could indeed skip the migration and just complain loudly about not being able to install the snap.

The definition of "unused" here would be whether LXD was ever activated since only the migration tool can properly analyse the database to determine whether it's worth performing a migration and that tool comes from the snap.

So the check would effectively be for server.crt and server.key, if they're not present, then assume the system was never activated and relax the snap install requirement.

[Stéphane Graber (stgraber)]

We still wouldn't be using the lxd-installer, instead the built-in command wrappers for lxc and lxd should just be made to fail with an error indicating that LXD should be manually installed.

Automatic installation on first use is something that's unlikely to be very supportable as it wouldn't offer a choice of channel/track for the LXD snap, nor would work with the variety of downstreams that are directly interacting with LXD through API.

[Dimitri John Ledkov (xnox)]

lxd client is usable; the lxd server is not under wsl, so it wouldn't be possible to complete lxd init under wsl.

do-release-upgrade should be usable under wsl; as it's no different to a chroot or like something like virtuozzo sans ability to run systemd.

[Stéphane Graber (stgraber)]

Yeah, so that's going to be a bit of an issue for some users as without snapd working in this environment, they'll actually loose the lxd client as part of the upgrade.

Whether that's an actual issue when the lxd client works natively on Windows, I don't really know.

[Stéphane Graber (stgraber)]

It feels like a lot of this pain comes from the fact that we are pushing an Ubuntu Cloud image to those users when large parts of that image are not going to be functional due to lack of systemd or some kernel APIs.

A better option would be to strip the WSL image down to a set of package that's more reasonable for use in such a chroot environment, removing systemd, removing all daemons (LXD included), ...
That will make the image much lighter and avoid such issues on upgrade.

[Balint Reczey (rbalint)]

I'm working towards stripping the WSL image a bit, and the first step is adding ubuntu-wsl package (LP: #1822341) to pull back things when needed.

The lxd client can be very useful for WSL users, since they can connect to remote LXD servers without the need for a VM or an other machine.

[Balint Reczey (rbalint)]

The proposed fix: https://github.com/lxc/lxd-pkg-ubuntu/pull/33

[Balint Reczey (rbalint)]

Please SRU to Cosmic as well.

[Stéphane Graber (stgraber)]

Upload to disco and cosmic.

@Balint can you make sure this bug has the suitable SRU paperwork sorted?

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxd - 1:0.6

---------------
lxd (1:0.6) disco; urgency=medium

  * Tweak debconf messages a bit.

lxd (1:0.5) disco; urgency=medium

  [ Balint Reczey ]
  * Allow skipping snap installation when LXD was never used
    (LP: #1821924)

  [ Stéphane Graber ]
  * Bump fallback Ubuntu release to 19.04

 -- Stéphane Graber <email address hidden> Fri, 05 Apr 2019 18:02:01 -0400

[Łukasz Zemczak (sil2100)]

Hello Balint, or anyone else affected,

Accepted lxd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxd/1:0.4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Balint Reczey (rbalint)]

Tested 1:0.4.1 on Cosmic:

Skipping the snap installation on upgrade works, but the installed command does not print an nice error message.

ubuntu@DESKTOP-V4DDKSS:~$ sudo do-release-upgrade
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [819 B]
Get:2 Upgrade tool [1250 kB]
Fetched 1251 kB in 0s (0 B/s)
authenticate 'cosmic.tar.gz' against 'cosmic.tar.gz.gpg'
extracting 'cosmic.tar.gz'
[screen is terminating]
ubuntu@DESKTOP-V4DDKSS:~$ dpkg -l lxd | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-========================================
ii lxd 1:0.4.1 all Transitional package - lxd -> snap (lxd)
ubuntu@DESKTOP-V4DDKSS:~$ lxd
/usr/bin/lxd: 8: exec: /snap/bin/lxd: not found

[Brian Murray (brian-murray)]

Hello Balint, or anyone else affected,

Accepted lxd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxd/1:0.4.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Balint Reczey (rbalint)]

Verified 1:0.4.2 on Cosmic:

1.

ubuntu@DESKTOP-V4DDKSS:~$ sudo do-release-upgrade
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [819 B]
Get:2 Upgrade tool [1243 kB]
Fetched 1244 kB in 0s (0 B/s)
authenticate 'cosmic.tar.gz' against 'cosmic.tar.gz.gpg'
extracting 'cosmic.tar.gz'
[screen is terminating]
ubuntu@DESKTOP-V4DDKSS:~$ lxd

Command '/usr/bin/lxd' requires the lxd snap to be installed.
Please install it with:

snap install lxd

ubuntu@DESKTOP-V4DDKSS:~$

Also reinstalling does not allow skip when lxd data is present but removing it lets the user skip again.

2.

root@cc-lxd:~# dpkg -l lxd | cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-========================================
ii lxd 1:0.4 all Transitional package - lxd -> snap (lxd)
root@cc-lxd:~# apt install -y -qq lxd
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following packages will be upgraded:
  lxd
1 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 11.4 kB of archives.
After this operation, 3072 B of additional disk space will be used.
Preconfiguring packages ...
(Reading database ... 28289 files and directories currently installed.)
Preparing to unpack .../archives/lxd_1%3a0.4.2_all.deb ...
=> Installing the LXD snap
==> Checking connectivity with the snap store
==> Installing the LXD snap from the latest track for ubuntu-18.10
snap "lxd" is already installed, see 'snap help refresh'
=> Snap installation complete
==> Cleaning up leftovers
Failed to stop lxd.socket: Unit lxd.socket not loaded.
Failed to stop lxd.service: Unit lxd.service not loaded.
Failed to stop lxd-containers.service: Unit lxd-containers.service not loaded.
Failed to disable unit: Unit file lxd.socket does not exist.
Unpacking lxd (1:0.4.2) over (1:0.4) ...
Setting up lxd (1:0.4.2) ...

3.

root@cc-lxd:~# snap remove lxd
lxd removed
root@cc-lxd:~# lxd

Command '/usr/bin/lxd' requires the lxd snap to be installed.
Please install it with:

snap install lxd

root@cc-lxd:~# lxc

Command '/usr/bin/lxc' requires the lxd snap to be installed.
Please install it with:

snap install lxd

root@cc-lxd:~#

4.

root@cc-lxd:~# apt purge lxd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The foll...

[Balint Reczey (rbalint)]

Autopkgtest is failing in cosmic but it passed for me locally in qemu.
Looks like an infra issue on amd64.

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxd - 1:0.4.2

---------------
lxd (1:0.4.2) cosmic; urgency=medium

  [ Balint Reczey ]
  * Fix error message on missing snap. (LP: #1821924)

lxd (1:0.4.1) cosmic; urgency=medium

  [ Balint Reczey ]
  * Allow skipping snap installation when LXD was never used
    (LP: #1821924)

  [ Stéphane Graber ]
  * Tweak debconf messages a bit.

 -- Stéphane Graber <email address hidden> Thu, 11 Apr 2019 10:37:34 -0400

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for lxd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.