No longer possible to use do-release-upgrade on Groovy

[Brian Murray]

Impact
------
It is not possible for users still running Ubuntu 20.10, an end of life release, to upgrade to a supported release of Ubuntu.

Test Plan
---------
G to I Test
-----------
On an Ubuntu 20.10 run `do-release-upgrade` and observe an error message about "an upgrade from Groovy to Impish is not supported".

With the version of ubuntu-release-upgrader from -proposed (accessible by running `do-release-upgrade -p`) the upgrade will be supported and complete successfully.

G to J Test
-----------
On an Ubuntu 20.10 system run `do-release-upgrade -d` and observe an error message about "an upgrade from Groovy to Jammy is not supported".

Now download the tarball from http://archive.ubuntu.com/ubuntu/dists/jammy-proposed/main/dist-upgrader-all/current/ and extract it to a subdirectory in /tmp/ and execute `jammy`. The upgrade should now proceed and you should not encounter any errors upgrading to Ubuntu 22.04 LTS.

H to J Test
-----------
Same as the above but replace Groovy with Hirsute.

Regression Potential
--------------------
Its possible that the upgrade process could fail and users will be left with a broken system instead of an insecure one. However, we'll utilize our automated upgrade testing of Ubuntu desktop (and flavors) to ensure the process works.

Additionally, the script utils/demotions.py was updated to support falling back from archive.u.c to old-releases.u.c in the event the Release file is not found at the former. Given that pre-build.sh generates the demotions file for impish and the demoted.cfg.impish file was not updated I think there is no chance of regression. in this part of the update.
SRU Note
--------
This'll also need fixing in Jammy so people still running Groovy could upgrade to it when Impish reaches its EoL, however that isn't really testable yet so will be fixed in Jammy later.

Original Description
--------------------
I tried running `do-release-upgrade` on an Ubuntu 20.10 (Groovy Gorilla) system and received an error message from dist-upgrader tarball that an upgrade from Groovy to Impish is not supported. This is because DistUpgrade.cfg (distributed by the tarball) contains `From=Hirsute` and `To=Impish`.

A minimal fix for this would be to create a DistUpgrade.cfg.groovy and ensure `From=Groovy`. You'd want to look at the contents of DistUpgrade.cfg and DistUpgrade.cfg.focal to sort out what DistUpgrade.cfg.groovy should contain. We might also want to create a new demoted.cfg file but the demoted.cfg.focal file might capture everything that was demoted. (For what its worth I tested an upgrade of a stock Ubuntu 20.10 system to Ubuntu 21.10 by copying DistUpgrade.cfg.focal to DistUpgrade.cfg.groovy and did not run into any issues.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mikolaj Buchwald (mikbuch)]

@Brian Murray, first, thank you so much for confirming the issue and creating this bug report.

Just to be clear, I finally was able to upgrade from groovy to impish by following the instructions in this post: https://blog.invid.eu/2022/02/24/upgrade-eol-ubuntu-server-from-20-10-to-21-10-an-upgrade-from-groovy-to-impish-is-not-supported-with-this-tool-ubuntu-ubuntugeek-ubuntu-linux-eol-upgrade/ ; see also the official documentation: https://help.ubuntu.com/community/EOLUpgrades

To the people new to this thread: see also my comment (https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1745754/comments/6) in the other bug report: https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1745754 ; and my notes on the issue: https://gist.github.com/mikbuch/2c349e1281c69818b48893a460c5accb

@Brian Murray, your suggestion for the minimal fix is all OK, but I wonder if a more general mechanism could be implemented, to support upgrading from an EOL release X to a (EOL) release Y (as the `do-release-upgrade` already says/recognizes with which release there was a problem to which: "An upgrade from ‘groovy’ to ‘impish’ is not supported with this tool."). I don't fully grasp how the DistUpgrade.cfg work, maybe my issue is/was related only to groovy. Either way, if the current bug will be resolved, maybe it is also good idea to update the documentation on EOL upgrades, as well (https://help.ubuntu.com/community/EOLUpgrades).

I can help, and I will, as soon as I will get more familiar with the `do-release-upgrade` process.

[Mikolaj Buchwald (mikbuch)]

@Brian Murray, if you don't mind, I will post my notes here, as comments, so that it will be easier for someone else to follow my thought process on getting familiar with the problem, i.e., getting to better know how does `do-release-upgrade` and `DistUpgrade.cfg` work.

It is very important to know all recent Ubuntu release names (https://wiki.ubuntu.com/Releases) in order to keep track with the analysis below.

Ok, so I inspected the `DistUpgrade.cfg` file (https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg?h=ubuntu%2Fjammy#n109; stored in my file system at `/usr/share/ubuntu-release-upgrader/DistUpgrade.cfg`). It contains:

...
108 [Sources]
109 From=impish
110 To=jammy
...

So it represents my most recent update. (I finally did groovy=>impish, and impish=>jammy.) Then, during the next update, it would look like:

...
108 [Sources]
109 From=jammy
110 To=kinetic
...

(see: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg#n110)

I also found the `DistUpgrade.cfg.focal` (https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg.focal), and it contains:

...
108 [Sources]
109 From=focal
110 To=jammy
...

Interestingly, the `DistUpgrade.cfg.focal` stored at my file system (at "~/.local/share/Trash/files/DistUpgrade.cfg.focal") contains:

...
108 [Sources]
109 From=focal
110 To=hirsute
...

E.g., it was moved to the trash when the `ubuntu-hirsute` branch was in use on my system, see: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg.focal?h=ubuntu%2Fhirsute#n109 .

---

@Brian Murray, your other suggestion was to create `demoted.cfg.groovy`. Indeed, in the https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/utils there is no `demoted.cfg.groovy`.

Maybe part of the simplest solution (for the others to use) is, as you suggest, to create "demoted.cfg.groovy" which contains basically the same packages as "demoted.cfg.focal".

---

Brian Murray wrote:

 > "For what its worth I tested an upgrade of a stock Ubuntu 20.10 system to Ubuntu 21.10 by copying DistUpgrade.cfg.focal to DistUpgrade.cfg.groovy and did not run into any issues"

It worked because of this piece of code: lines 44 & 45 here: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeConfigParser.py#n44
(
```
if os.path.exists(maincfg + "." + from_release):
    maincfg += "." + from_release
```
)

It is not a particularly neat solution, but it will work.

In the ideal world, there would be a mechanism similar to this: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeConfigParser.py#n39 (from_release = ...) in combination with the list of known releases (https://wiki.ubuntu.com/Releases) for the `DistUpgradeConfigParser.py` not to take fixed release names (from `DistUpgrade.cfg`/focal/groovy), but dynamically decide what is the release on the system, and what is the next available release, so that the `do-release-upgrade` would attempt to upgrade to the next available release. This way upgrades from the EOL systems would be supported, or...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mikolaj Buchwald (mikbuch)]

Hi, @Brian Murray, I tested the solution you suggested on a newly-installed Ubuntu 20.10 Groovy, and it worked. I described the steps to reproduce the problem, and the solution here: https://gist.github.com/mikbuch/2c349e1281c69818b48893a460c5accb?permalink_comment_id=4206418#gistcomment-4206418

I also created a merge request (https://code.launchpad.net/~mikbuch/ubuntu/+source/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/425341), but I didn't know the patch version for the proper commit message. I am also not sure if I should have committed directly to ubuntu/impish, or rather, e.g., to the ubuntu/impish-devel. Hope that helps you somehow. Let me know if I could do something more to help you.

[Robie Basak (racb)]

@Brian data/demoted.cfg.{bionic,focal,trusty,xenial} symlinks exist but one isn't added for the new utils/demoted.cfg.groovy. Is this OK?

Accepting anyway as it's probably fine, but I thought I'd mention it.

[Robie Basak (racb)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:21.10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Using the new version of the dist-upgrader I was successfully able to upgrade from Groovy to Impish.

bdmurray@clean-groovy-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.10
DISTRIB_CODENAME=impish
DISTRIB_DESCRIPTION="Ubuntu 21.10"
bdmurray@clean-groovy-amd64:~$ head -n20 /var/log/dist-upgrade/main.log
2022-07-06 08:12:44,568 INFO Using config files '['./DistUpgrade.cfg.groovy', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgr
ades.cfg']'
2022-07-06 08:12:44,568 INFO uname information: 'Linux clean-groovy-amd64 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_
64'
2022-07-06 08:12:44,852 INFO apt version: '2.1.10ubuntu0.3'
2022-07-06 08:12:44,852 INFO python version: '3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 10.3.0]'
2022-07-06 08:12:44,854 INFO release-upgrader version '21.10.10' started
2022-07-06 08:12:44,858 INFO locale: 'en_US' 'UTF-8'
2022-07-06 08:12:44,941 DEBUG Using 'DistUpgradeViewText' view
2022-07-06 08:12:44,984 DEBUG enable dpkg --force-overwrite
2022-07-06 08:12:45,029 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
2022-07-06 08:12:47,118 DEBUG lsb-release: 'groovy'
2022-07-06 08:12:47,119 DEBUG _pythonSymlinkCheck run
2022-07-06 08:12:47,124 DEBUG openCache()
2022-07-06 08:12:47,124 DEBUG quirks: running PreCacheOpen
...
bdmurray@clean-groovy-amd64:~$ tail -n10 /var/log/dist-upgrade/main.log
2022-07-06 08:37:51,460 DEBUG 'linux-image-5.8.0-63-generic' scheduled for remove but not safe to remove, skipping
2022-07-06 08:37:52,139 DEBUG skipping running kernel pkg 'linux-headers-5.8.0-63-generic'
2022-07-06 08:37:52,140 DEBUG 'linux-headers-5.8.0-63-generic' scheduled for remove but not safe to remove, skipping
2022-07-06 08:37:53,112 DEBUG Finish checking for obsolete pkgs
2022-07-06 08:37:53,165 DEBUG The following packages are marked for removal: ca-certificates-java libgssapi3-heimdal default-jre-headless l
iblibreoffice-java liborcus-parser-0.15-0 default-jre libedataserverui-1.2-2 libwind0-heimdal libridl-java libldap-2.4-2 libboost-iostreams
1.71.0 realmd libgdk-pixbuf2.0-0 libatk-wrapper-java linux-image-5.8.0-25-generic libpython3.8-minimal vino libraw19 gcc-10-base libjuh-jav
a libheimntlm0-heimdal xul-ext-ubufox libtracker-sparql-2.0-0 libamtk-5-common perl-modules-5.30 libheimbase1-heimdal libpython3.8 python3.
8 linux-modules-extra-5.8.0-25-generic libbrlapi0.7 linux-headers-5.8.0-25 linux-headers-5.8.0-63 libboost-locale1.71.0 libappindicator3-1
ure-java libhandy-0.0-0 openjdk-11-jre-headless linux-headers-5.8.0-63-generic libisl22 libboost-filesystem1.71.0 libtepl-5-0 lz4 linux-hea
ders-5.8.0-25-generic libtracker-miner-2.0-0 libatk-wrapper-java-jni sssd-dbus adcli libperl5.30 libllvm11 libsnmp35 libboost-date-time1.71
.0 libboost-thread1.71.0 libjurt-java libfwupdplugin1 linux-modules-5.8.0-25-generic python3.8-minimal openjdk-11-jre ubuntu-wallpapers-gro
ovy libhcrypto4-heimdal fonts-dejavu-extra sssd-tools liborcus-0.15-0 mysql-common libmysqlclient21 libphonenumber7 libpython3.8-stdlib lib
tracker-control-2.0-0 libroken18-heimdal libpoppler102 firefox gdbserver libasn1-8-heimdal libkrb5-26-heimdal python3-requests-unixsoc...

[Brian Murray (brian-murray)]

Additionally automatic upgrade testing from Groovy to Impish (for Ubuntu and flavors) is succeeding and the results are viewable to Canonical employees only (sorry!) here:

https://platform-qa-jenkins.ubuntu.com/view/Upgrade/

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:21.10.10

---------------
ubuntu-release-upgrader (1:21.10.10) impish; urgency=medium

  * Ship a DistUpgrade.cfg.groovy and demoted.cfg.groovy to allow upgrades
    from Groovy to Impish to be allowed. (LP: #1975533)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

 -- Brian Murray <email address hidden> Mon, 27 Jun 2022 15:47:00 -0700

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:22.04.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-release-upgrader (1:22.04.12) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:22.04.12 (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-release-upgrader

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Brian Murray (brian-murray)]

The attached screenshot shows a successful upgrade from Groovy to Jammy!

bdmurray@clean-groovy-amd64:~$ head /var/log/dist-upgrade/main.log
2022-07-22 12:37:59,572 INFO Using config files '['./DistUpgrade.cfg.groovy', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgrades.cfg']'
2022-07-22 12:37:59,572 INFO uname information: 'Linux clean-groovy-amd64 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64'
2022-07-22 12:37:59,881 INFO apt version: '2.1.10ubuntu0.3'
2022-07-22 12:37:59,881 INFO python version: '3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 10.3.0]'
2022-07-22 12:37:59,883 INFO release-upgrader version '22.04.12' started
2022-07-22 12:37:59,888 INFO locale: 'en_US' 'UTF-8'
2022-07-22 12:37:59,970 DEBUG Using 'DistUpgradeViewText' view
2022-07-22 12:38:00,024 DEBUG enable dpkg --force-overwrite
2022-07-22 12:38:00,067 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'

[Brian Murray (brian-murray)]

With the version of ubuntu-release-upgrader in -proposed I was also able to conduct a successful upgrade from Hirsute to Jammy.

bdmurray@clean-hirsute-amd64:/tmp/uru$ sudo ./jammy --frontend DistUpgradeViewText
[screen is terminating]
bdmurray@clean-hirsute-amd64:/tmp/uru$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
bdmurray@clean-hirsute-amd64:/tmp/uru$ head /var/log/dist-upgrade/main.log
2022-07-22 18:18:41,143 INFO Using config files '['./DistUpgrade.cfg.hirsute', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgrades.cfg']'
2022-07-22 18:18:41,144 INFO uname information: 'Linux clean-hirsute-amd64 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64'
2022-07-22 18:18:41,405 INFO apt version: '2.2.4ubuntu0.1'
2022-07-22 18:18:41,406 INFO python version: '3.9.5 (default, Nov 18 2021, 16:00:48)
[GCC 10.3.0]'
2022-07-22 18:18:41,408 INFO release-upgrader version '22.04.12' started
2022-07-22 18:18:41,411 INFO locale: 'en_US' 'UTF-8'
2022-07-22 18:18:41,484 DEBUG Using 'DistUpgradeViewText' view
2022-07-22 18:18:41,542 DEBUG enable dpkg --force-overwrite
2022-07-22 18:18:41,591 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:22.04.12

---------------
ubuntu-release-upgrader (1:22.04.12) jammy; urgency=medium

  [ Brian Murray ]
  * Add support for upgrading from End of Life releases (Ubuntu 20.10 and
    Ubuntu 21.04) to Ubuntu 22.04. (LP: #1975533)
  * DistUpgrade: Do not attempt to reboot in WSL. (LP: #1958668)
  * DistUpgrade/deb2snap.json: gnome-3-34-1804 is no longer a seeded snap and
    should not be refreshed. (LP: #1981485)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

  [ William 'jawn-smith' Wilson ]
  * tests/test_quirks.py: Remove declaration of unused variables

 -- Brian Murray <email address hidden> Tue, 19 Jul 2022 06:52:28 -0700

[Mikolaj Buchwald (mikbuch)]

Great news, thank you for the work on this issue!