ubuntu-release-upgrader should transition debs to snaps

[Brian Murray]

For new installs the desktop version of Ubuntu 18.04 includes snap apps of some utilities[1]. To ensure that there is little difference between a system upgrading between releases of Ubuntu and fresh installs ubuntu-release-upgrader should be modified to install the snap format of those utilities and remove the deb format package. This should only be done for Ubuntu Desktop installs and if the system has a connection to the Snap Store.

[Test Case]
1) Install an Ubuntu desktop system with Ubuntu 16.04
2) Upgrade to Ubuntu 18.04
3) Run 'snap list'

With the version of ubuntu-release-upgrader from -proposed you should see snaps installed for gtk-common-themes, gnome-calculator, gnome-characters, gnome-logs, gnome-system-monitor. Additionally, the deb version of those four packages should be removed, although if one of the snaps failed to install then the corresponding deb would not be removed.

It also be good to test that it is still possible to upgrade without a connection to the Snap store and to test this only happens on a system with ubuntu-desktop installed.

[Regression Potential]
Its possible that the snaps could be install on a system without ubuntu-desktop so we should explicitly test that scenario. The snaps installed should be tested briefly to ensure they are launchable and we don't leave upgraders without a calculator.

[Note]
This has been done in a DistUpgradeQuirk and will look different for Cosmic than Bionic as the function name includes the release i.e. cosmicPostUpgrade vs bionicPostUpgrade.

[1] https://wiki.ubuntu.com/BionicBeaver/ReleaseNotes#Ubuntu_Desktop

[Brian Murray (brian-murray)]

ubuntu-release-upgrader (1:18.10.4) cosmic; urgency=medium

  * DistUpgradeQuirks.py, DistUpgradeController.py: move inhibiting of
    gnome-session idle so that it is called immediately after the upgrade
    starts (LP: #1778817), present a dialog notifying that the lock screen has
    been disabled (LP: #1174093), and do not reenable the lock screen as it
    may fail to work after a release upgrade. (LP: #1780342)
  * DistUpgradeQuirks.py: Add in a quirk for upgrading to Cosmic that will
    check if the system has connectivity to the snap store and replace some
    desktop packages with snaps to ensure the system is similar to a fresh
    install.

 -- Brian Murray <email address hidden> Fri, 06 Jul 2018 11:21:11 -0700

[Łukasz Zemczak (sil2100)]

Since this fix involves a lot of new logic being implemented, I would like the validation to include *all* the aforementioned test cases: regular desktop upgrade, server upgrade (making sure no snaps are installed), desktop upgrade with no snap store connectivity (or credentials?). I would also like to see a quick test for cases when, let's say, one of these apps has both snap and deb versions on the system before upgrade.

I'm also slightly concerned about the case handling in _checkStoreConnectivity(). The res variable is not defined outside of the conditionals, and I was wondering if there is a possibility of `snap debug connectivity` returning an error that we did not expect (especially since there is no else: case handled). In that case u-r-u will just crash due to res being undefined.

Brian, I will be accepting it as is, but if that's indeed a valid concern I'd prefer if this was fixed with an upload on top. If not, we're good as is.

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Tested an upgrade from Ubuntu 16.04 to Ubuntu 18.04:

bdmurray@clean-xenial-amd64:/tmp/u-r-u$ head /var/log/dist-upgrade/main.log
2018-07-11 15:30:56,243 INFO Using config files '['./DistUpgrade.cfg.xenial']'
2018-07-11 15:30:56,243 INFO uname information: 'Linux clean-xenial-amd64 4.4.0-121-generic #145-Ubuntu SMP Fri Apr 13 13:47:23 UTC 2018 x86_64'
2018-07-11 15:30:56,530 INFO apt version: '1.2.26'
2018-07-11 15:30:56,531 INFO python version: '3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609]'
2018-07-11 15:30:56,534 INFO release-upgrader version '18.04.20' started
2018-07-11 15:30:56,537 INFO locale: 'en_US' 'UTF-8'
2018-07-11 15:30:56,756 DEBUG Using 'DistUpgradeViewText' view
2018-07-11 15:30:56,801 DEBUG enable dpkg --force-overwrite
2018-07-11 15:30:56,834 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
bdmurray@clean-xenial-amd64:/tmp/u-r-u$ snap list
Name Version Rev Tracking Developer Notes
core 16-2.33.1 4917 stable canonical core
gnome-3-26-1604 3.26.0 70 stable canonical -
gnome-calculator 3.28.2 180 stable/… canonical -
gnome-characters 3.28.2 103 stable/… canonical -
gnome-logs 3.28.2 37 stable/… canonical -
gnome-system-monitor 3.28.2 51 stable/… canonical -
gtk-common-themes 0.1 319 stable/… canonical -
bdmurray@clean-xenial-amd64:/tmp/u-r-u$ for p in gnome-calculator gnome-characters gnome-logs gnome-system-monitor; do apt-cache policy $p; done
gnome-calculator:
  Installed: 1:3.28.1-1ubuntu1
  Candidate: 1:3.28.1-1ubuntu1
  Version table:
 *** 1:3.28.1-1ubuntu1 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status
gnome-characters:
  Installed: (none)
  Candidate: 3.28.2-0ubuntu1
  Version table:
     3.28.2-0ubuntu1 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
     3.28.0-3 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
gnome-logs:
  Installed: (none)
  Candidate: 3.28.0-1
  Version table:
     3.28.0-1 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages
gnome-system-monitor:
  Installed: (none)
  Candidate: 3.28.2-0ubuntu1
  Version table:
     3.28.2-0ubuntu1 500
        500 http://192.168.10.7/ubuntu bionic-updates/main amd64 Packages
     3.28.1-1 500
        500 http://192.168.10.7/ubuntu bionic/main amd64 Packages

The fact that gnome-calculator was not removed was expected because unity-scope-calculator depends on it. From /var/log/dist-upgrade/main.log:

2018-07-11 16:55:42,517 DEBUG Start checking for obsolete pkgs
2018-07-11 16:56:07,402 DEBUG package 'gnome-calculator' has unwanted removals, skipping
2018-07-11 16:56:11,692 DEBUG 'gnome-calculator' scheduled for remove but not safe to remove, skipping
2018-07-11 16:56:26,776 DEBUG Finish checking for obsolete pkgs

This completes the main verification of the SRU there is still some additional testing needed though.

[Brian Murray (brian-murray)]

Testing an upgrade from Ubuntu 16.04 to Ubuntu 18.04 where I modified /etc/hosts so api.snapcraft.io resolved to 127.0.0.1.

"Connection to Snap Store failed

Your system does not have a connection to the Snap Store. For the
best upgrade experience make sure that your system can connect to
api.snapcraft.io.
Do you still want to continue with the upgrade?

Continue [yN] y"

So the dialog about the lack of a connection was displayed but later on I got a crash which is found in bug 1781412.

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I performed the verification again with the dist-upgrader version 18.04.21 to ensure there wasn't a regression in my DistUpgradeQuirks changes for bug 1781412.

2018-07-16 14:46:21,478 DEBUG Install of snap gtk-common-themes succeeded
2018-07-16 14:47:00,940 DEBUG Install of snap gnome-calculator succeeded
2018-07-16 14:47:09,314 DEBUG Install of snap gnome-characters succeeded
2018-07-16 14:47:17,970 DEBUG Install of snap gnome-logs succeeded
2018-07-16 14:47:27,709 DEBUG Install of snap gnome-system-monitor succeeded
bdmurray@clean-xenial-amd64:/tmp/u-r-u$ snap list
Name Version Rev Tracking Developer Notes
core 16-2.33.1 4917 stable canonical core
gnome-3-26-1604 3.26.0 70 stable canonical -
gnome-calculator 3.28.2 180 stable/… canonical -
gnome-characters 3.28.2 103 stable/… canonical -
gnome-logs 3.28.2 37 stable/… canonical -
gnome-system-monitor 3.28.2 51 stable/… canonical -
gtk-common-themes 0.1 319 stable/… canonical -
bdmurray@clean-xenial-amd64:/tmp/u-r-u$ cat DistUpgradeVersion.py
VERSION = '18.04.21'

[Brian Murray (brian-murray)]

I happened to be testing a distribution upgrade of Lubuntu 16.04 to Lubuntu 18.04 and can confirm that no snaps were installed during the upgrade when using ubuntu-release-upgrader version 18.04.21.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.21

---------------
ubuntu-release-upgrader (1:18.04.21) bionic; urgency=medium

  * DistUpgradeQuirks.py: when a snap fails to install try the next one
    instead of trying to check the return code of the called process which
    causes a Traceback. (LP: #1781412)
  * data/mirrors.cfg: Update the list of official mirrors.

ubuntu-release-upgrader (1:18.04.20) bionic; urgency=medium

  * Purge obsolete packages, if user agrees to remove obsolete
    packages. LP: #1775660
  * DistUpgradeQuirks.py, DistUpgradeController.py: move inhibiting of
    gnome-session idle so that it is called immediately after the upgrade
    starts (LP: #1778817), present a dialog notifying that the lock screen has
    been disabled (LP: #1174093), and do not reenable the lock screen as it
    may fail to work after a release upgrade. (LP: #1780342)
  * DistUpgradeQuirks.py: Add in a quirk for upgrading to Bionic that will
    check if the system has connectivity to the snap store and replace some
    desktop packages with snaps to ensure the system is similar to a fresh
    install. (LP: #1780841)
  * data/mirrors.cfg: Update the list of official mirrors.

 -- Brian Murray <email address hidden> Sun, 15 Jul 2018 10:16:16 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Mark Eichin (eichin-gmail)]

Not sure if this should get a new ticket, but if `snap debug connectivity` fails, the whole upgrade mysteriously aborts (user sees ```Reading state information... Done

Restoring original system state

Aborting
``` and `main.log` shows ```2018-12-12 21:24:20,067 DEBUG running Quirks.bionicPostInitialUpdate
2018-12-12 21:24:20,367 DEBUG abort called
``` with literally no other output anywhere.

`strace` is what finds the `snap debug connectivity` call, and `service snapd status` shows `Active: failed (Result: start-limit-hit)` - `service snapd start` fixes that and `do-release-upgrade` goes through fine.

(It's not just me, https://askubuntu.com/questions/1078692/cant-upgrade-ubuntu-16-04-to-18-04-lts/1100514 looks to be the same symptoms, but it wasn't until I'd figured out that `snap` was involved that I found this feature-ticket; I don't think it's widely known, I believe most users don't even know snap is *in* 16.04, and certainly won't notice `snapd` being broken without other hints...)

[Brian Murray (brian-murray)]

On Thu, Dec 13, 2018 at 03:50:51AM -0000, Mark Eichin wrote:
> Not sure if this should get a new ticket, but if `snap debug
> connectivity` fails, the whole upgrade mysteriously aborts (user sees
> ```Reading state information... Done
>
> Restoring original system state
>
> Aborting
> ``` and `main.log` shows ```2018-12-12 21:24:20,067 DEBUG running Quirks.bionicPostInitialUpdate
> 2018-12-12 21:24:20,367 DEBUG abort called
> ``` with literally no other output anywhere.
>
> `strace` is what finds the `snap debug connectivity` call, and `service
> snapd status` shows `Active: failed (Result: start-limit-hit)` -
> `service snapd start` fixes that and `do-release-upgrade` goes through
> fine.
>
> (It's not just me, https://askubuntu.com/questions/1078692/cant-upgrade-
> ubuntu-16-04-to-18-04-lts/1100514 looks to be the same symptoms, but it
> wasn't until I'd figured out that `snap` was involved that I found this
> feature-ticket; I don't think it's widely known, I believe most users
> don't even know snap is *in* 16.04, and certainly won't notice `snapd`
> being broken without other hints...)

Could you please open a new bug report about the issue you've
encountered using 'ubuntu-bug ubuntu-release-upgrader-core'? That'll
include log files which might help.

Thanks!
--
Brian Murray