ubuntu-release-upgrader should purge obsolete packages, not just remove them

[Steve Langasek]

[Justification]
When an obsolete package leaves conffiles behind, these conffiles can sometimes interact poorly with the system and cause bugs. Since the obsoleted package no longer exists in the targeted release, this can be difficult to resolve by applying fixes to a package. Furthermore, removing-but-not-purging packages on upgrade means that a system which is upgraded will as a rule be inconsistent with a newly-installed system, which should always be considered a bug.

Wherever possible, we should strive to make an upgraded system look exactly like a newly installed system, to minimize the combinatorics of configurations that must be supported.

Examples of cases where this has been a problem in practice:

LP: #1770965
LP: #1749199

[Test case]
1. Do an upgrade from xenial with the release ubuntu-release-upgrader. Accept the option to remove obsolete packages.
2. Capture the list of packages that are in state 'rc' afterwards.
3. Do an equivalent upgrade with the -proposed ubuntu-release-upgrader, including removal of obsolete packages.
4. Confirm that the upgrade completes successfully.
5. Capture the list of packages that are in state 'rc' afterwards.
6. Confirm that there are fewer packages in state 'rc' in the second upgrade than the first.
7. Repeat all of the above with an upgrade from artful.

[Regression potential]
As a consequence of this, we will also be running a lot more postrm scripts with the 'purge' option on users desktops, so any bugs in those maintainer scripts will now be apparent as part of a release upgrade where before they were latent.

[Brian Murray (brian-murray)]

ubuntu-release-upgrader (1:18.10.3) cosmic; urgency=medium

  [ Dimitri John Ledkov ]
  * Purge obsolete packages, if user agrees to remove obsolete
    packages. LP: #1749199
  * Make dist-upgrade.py work when PWD != datadir, by fixing the logic for
    finding and loading .release config files with datadir prepended
    first.

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeCache.py: clarify why a package is not removed.

 -- Brian Murray <email address hidden> Wed, 27 Jun 2018 16:34:25 -0700

[Łukasz Zemczak (sil2100)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I performed an upgrade of an Ubuntu 16.04 desktop to Ubuntu 18.04 today and ran "dpkg -l | grep '^rc'" after purging packages and captured the difference.

 $ diff --new-line-format="" --unchanged-line-format="" xenial-nopurge.txt xenial-purge.txt
rc gnome-user-share 3.18.3-3 amd64 User level public file sharing via WebDAV
rc initscripts 2.88dsf-59.3ubuntu2 amd64 scripts for initializing and shutting down the system
rc insserv 1.14.0-5ubuntu3 amd64 boot sequence organizer using LSB init.d script dependency information
rc libhybris 0.1.0+git20151016+6d424c9-0ubuntu7 amd64 Allows to run bionic-based HW adaptations in glibc systems - libs
rc mountall 2.54ubuntu1 amd64 filesystem mounting tool
rc signond 8.59+17.10.20170606-0ubuntu1 amd64 Single Sign On framework
rc snap-confine 2.32.9+18.04 amd64 Transitional package for snapd
rc suru-icon-theme 16.10+18.04.20180421.1-0ubuntu1 all Ubuntu Suru Icon theme
rc sysv-rc 2.88dsf-59.3ubuntu2 all System-V-like runlevel change mechanism
[ 1:27PM 10767 ] [ bdmurray@impulse:~/tmp/bug-1775660 ]
 $ wc -l xenial-nopurge.txt
51 xenial-nopurge.txt
[ 1:29PM 10768 ] [ bdmurray@impulse:~/tmp/bug-1775660 ]
 $ wc -l xenial-purge.txt
42 xenial-purge.txt

So this looks good to me for an upgrade from Ubuntu 16.04.

[Brian Murray (brian-murray)]

Oh it might be worth noting I had to manually download the -proposed version of the dist-upgrader for bionic because you can't use 'do-release-upgrade -d -p' since there is no meta-release-development-proposed.

[Brian Murray (brian-murray)]

I performed an upgrade of Ubuntu 17.10 to Ubuntu 18.04 and ran "dpkg -l | grep '^rc'" after purging packages and kind of captured the difference in that with -proposed no packages were left in an rc state.

 $ cat artful-nopurge.txt
rc gnome-orca 3.28.0-3ubuntu1 all Scriptable screen reader - transitional package
rc libunistring0:amd64 0.9.3-5.2ubuntu1 amd64 Unicode string library for C
rc linux-image-4.13.0-16-generic 4.13.0-16.19 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-21-generic 4.13.0-21.24 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-41-generic 4.13.0-41.46 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-4.13.0-43-generic 4.13.0-43.48 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-16-generic 4.13.0-16.19 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-21-generic 4.13.0-21.24 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-41-generic 4.13.0-41.46 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc linux-image-extra-4.13.0-43-generic 4.13.0-43.48 amd64 Linux kernel extra modules for version 4.13.0 on 64 bit x86 SMP
rc python 2.7.15~rc1-1 amd64 interactive high-level object-oriented language (default version)
rc python2.7-minimal 2.7.15~rc1-1 amd64 Minimal subset of the Python language (version 2.7)
rc snapd-login-service 1.23-0ubuntu1 amd64 Daemon to allow non-root access to snapd

bdmurray@clean-artful-amd64:~$ grep "upgrader version" /var/log/dist-upgrade/main.log
2018-07-11 14:09:34,978 INFO release-upgrader version '18.04.20' started
bdmurray@clean-artful-amd64:~$ dpkg -l | grep "^rc"
bdmurray@clean-artful-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04

[Łukasz Zemczak (sil2100)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:18.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

The 18.04.21 upload just changes DistUpgradeQuirks for the deb to snap transition so I'm resetting the tag to verification done based of my test with 18.04.20.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:18.04.21

---------------
ubuntu-release-upgrader (1:18.04.21) bionic; urgency=medium

  * DistUpgradeQuirks.py: when a snap fails to install try the next one
    instead of trying to check the return code of the called process which
    causes a Traceback. (LP: #1781412)
  * data/mirrors.cfg: Update the list of official mirrors.

ubuntu-release-upgrader (1:18.04.20) bionic; urgency=medium

  * Purge obsolete packages, if user agrees to remove obsolete
    packages. LP: #1775660
  * DistUpgradeQuirks.py, DistUpgradeController.py: move inhibiting of
    gnome-session idle so that it is called immediately after the upgrade
    starts (LP: #1778817), present a dialog notifying that the lock screen has
    been disabled (LP: #1174093), and do not reenable the lock screen as it
    may fail to work after a release upgrade. (LP: #1780342)
  * DistUpgradeQuirks.py: Add in a quirk for upgrading to Bionic that will
    check if the system has connectivity to the snap store and replace some
    desktop packages with snaps to ensure the system is similar to a fresh
    install. (LP: #1780841)
  * data/mirrors.cfg: Update the list of official mirrors.

 -- Brian Murray <email address hidden> Sun, 15 Jul 2018 10:16:16 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.