do-release-upgrade crashed with KeyError in __getitem__(): "The cache has no package named 'python-is-python2'"

[Brian Murray]

[Impact]
Users with universe not enabled and python2 packages installed are unable to upgrade from Ubuntu 18.04 to Ubuntu 20.04.

[Test Plan]
-----------
On an Ubuntu 18.04 system with universe and multiverse disabled
1) apt-get install duplicity (it depends on python2.7)
2) run do-release-upgrade

Observe the upgrader hang at Calculating the changes (in text mode) kill it and observe a Traceback in /var/log/dist-upgrade/main.log.

...
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 303, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'python-is-python2'"

With the version of the release-upgrader from -proposed (do-release-upgrade -p) you'll instead receive a message regarding universe not being enabled and the need to either remove python2 packages or enable universe.

[Where problems could occur]
The switch to using the finally clause in DistUpgradeCache.py is a significant (albeit safe) change but it is necessary for the error message to actually be displayed. So it'd make sense to run through a whole release upgrade to ensure that the gui behaves normally.

[Original Description]
This is because python-minimal was installed and universe was not enabled on the system being upgraded.

Error Tracker buckets:
https://errors.ubuntu.com/problem/f92900f3c057db7600d88857e7c05434b69d2496
https://errors.ubuntu.com/problem/23e6edb38993cb74331d30beeadccdfffe1d557a

ProblemType: CrashDistroRelease: Ubuntu 20.04
Package: ubuntu-release-upgrader-core 1:20.04.31
ProcVersionSignature: Ubuntu 5.4.0-71.79~18.04.1-generic 5.4.101
Uname: Linux 5.4.0-71-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.23
Architecture: amd64
CrashDB: ubuntu
Date: Thu Apr 15 10:59:53 2021
ExecutablePath: /usr/bin/do-release-upgrade
ExecutableTimestamp: 1606763386
InstallationDate: Installed on 2021-04-13 (1 days ago)
InstallationMedia: Ubuntu 18.04.5 LTS "Bionic Beaver" - Release amd64 (20200806.1)
InterpreterPath: /usr/bin/python3.6
PackageArchitecture: all
ProcCmdline: /usr/bin/python3 -s ./focal --frontend DistUpgradeViewText
ProcCwd: /tmp/ubuntu-release-upgrader-e397n4sn
ProcEnviron:
 TERM=screen
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 LC_CTYPE=en_US.UTF-8
 PATH=(custom, no user)
Python3Details: /usr/bin/python3.6, Python 3.6.9, python3-minimal, 3.6.7-1~18.04
PythonArgs: ['/usr/bin/do-release-upgrade', '--frontend', 'DistUpgradeViewText']
PythonDetails: /usr/bin/python2.7, Python 2.7.17, python-minimal, 2.7.15~rc1-1SourcePackage: ubuntu-release-upgrader
Title: do-release-upgrade crashed with KeyError in __getitem__(): "The cache has no package named 'python-is-python2'"
UpgradeStatus: Upgraded to bionic on 2021-04-15 (0 days ago)

[Brian Murray (brian-murray)]

The error messaging in this scenario should be more informative.

[Matthias Klose (doko)]

this could be easily worked around by enabling universe during the upgrade

[Brian Murray (brian-murray)]

The same changes to the python is python2 quirk exist in the Impish version of ubuntu-release-upgrader although the quirk has been disabled as it is unnecessary.

https://launchpadlibrarian.net/546274717/ubuntu-release-upgrader_1%3A21.10.3_1%3A21.10.4.diff.gz

[Timo Aaltonen (tjaalton)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.34 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-release-upgrader (1:20.04.34) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:20.04.34 (armhf, ppc64el, arm64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#ubuntu-release-upgrader

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

I followed the described test case and received the following dialog after running 'do-release-upgrade -p' on an Ubuntu 18.04 system with universe disabled and python2 packages installed.

universe component not enabled

You have the package python-minimal installed which is a python2
package. python2 has been deprecated in Ubuntu 20.04 LTS and is now
available from the universe component of the Ubuntu archive. To
ensure any existing third party code you have installed continues to
work the release upgrade process would install python-is-python2 for
you. However, the universe component of the archive is not enabled on
this system. For the upgrade process to proceed you should either
enable the universe component in /etc/apt/sources.list or remove
python-minimal, libpython-stdlib

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command detached from window (Fri Jul 2 10:22:36 2021) ===
=== Command terminated with exit status 1 (Fri Jul 2 10:22:46 2021) ===

[Brian Murray (brian-murray)]

I also followed the described test case but using DistUpgradeViewGtk3 instead of DistUpgradeViewText. With the GUI version of the release upgrader I also received the notifcation regarding universe not being enabled.

[Brian Murray (brian-murray)]

Per the where problems could occur section I also ran a release upgrade using the GUI view w/o universe disabled and the succeeded.

bdmurray@clean-bionic-amd64:~$ head /var/log/dist-upgrade/main.log
2021-07-02 12:59:55,150 INFO Using config files '['./DistUpgrade.cfg.bionic', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgra
des.cfg']'
2021-07-02 12:59:55,150 INFO uname information: 'Linux clean-bionic-amd64 5.4.0-77-generic #86~18.04.1-Ubuntu SMP Fri Jun 18 01:23:22 UTC 20
21 x86_64'
2021-07-02 12:59:55,540 INFO apt version: '1.6.14'
2021-07-02 12:59:55,541 INFO python version: '3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0]'
2021-07-02 12:59:55,543 INFO release-upgrader version '20.04.34' started
2021-07-02 12:59:55,598 INFO locale: 'en_US' 'UTF-8'
2021-07-02 12:59:55,798 DEBUG Using 'DistUpgradeViewGtk3' view
2021-07-02 12:59:55,881 DEBUG enable dpkg --force-overwrite
2021-07-02 12:59:55,912 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
bdmurray@clean-bionic-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.35 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

ubuntu-release-upgrader version 1:20.04.35 just contained a test fix so I'm setting this back to verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.35

---------------
ubuntu-release-upgrader (1:20.04.35) focal; urgency=medium

  * tests/test_sources_list.py: remove a port architecture test as the code
    change has not been SRU'ed to 20.04.

ubuntu-release-upgrader (1:20.04.34) focal; urgency=medium

  * DistUpgrade/DistUpgradeQuirks.py: If universe is not enabled and the
    upgrader is trying to transition from python to python2 produce an error
    message and exit the upgrade process. (LP: #1924604)
  * DistUpgrade/DistUpgradeCache.py: switch to using finally in distUpgrade so
    that error messages in the above quirk will be visible.
  * tests/test_sources_list.py: use distro_info to set releases when
    performing some upgrade tests which fixes test failures when precise
    was moved to old-releases.
  * Update mirrors, demotions, and translations.

 -- Brian Murray <email address hidden> Fri, 02 Jul 2021 16:51:29 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.