18.04 to 20.04.1 upgrade on raspberry pi removes too many kernel meta packages

[Kleber Sacilotto de Souza]

Test Case
---------
1) On a Raspberry Pi running the 18.04 LTS image run "do-release-upgrade -d"
2) Once the upgrade completes observe that linux-raspi and linux-headers-raspi are offered for removal. (You'll see this when asked about removing obsolete packages.)

With the version of the dist-upgrader from -proposed (which you'll need to manually download, unpack, and run) you'll notice that linux-raspi and linux-headers-raspi are still installed after the upgrade completes.

Regression Potential
--------------------
The distribution upgrade code is now just not considering some additional packages for removal. This is a safe operation and if there was a regression it would be in poorly written python code which could cause a Traceback. So examine the actual change carefully.

Original Description
-------------------
In a raspberry pi system running 18.04, when running "do-release-upgrade -d" to upgrade to 20.04.1,
the transition from linux-raspi2-hwe-18.04 to linux-raspi doesn't seem to be done correctly and the system ends up without this and the linux-headers-raspi meta packages.

These are the meta packages installed before the release upgrade:

ii linux-headers-raspi2-hwe-18.04 5.3.0.1017.6 arm64 Linux kernel headers for Raspberry Pi systems.
ii linux-image-raspi2-hwe-18.04 5.3.0.1017.6 arm64 Linux kernel image for Raspberry Pi systems.
ii linux-raspi2-hwe-18.04 5.3.0.1017.6 arm64 Complete Linux kernel for Raspberry Pi systems.

After the upgrade is complete this is the only kernel meta installed:

ii linux-image-raspi 5.4.0.1015.50 arm64 Raspberry Pi Linux kernel image

During the upgrade, these kernel packages update are made:

Selecting previously unselected package linux-raspi-headers-5.4.0-1015.
Preparing to unpack .../244-linux-raspi-headers-5.4.0-1015_5.4.0-1015.15_arm64.deb ...
Unpacking linux-raspi-headers-5.4.0-1015 (5.4.0-1015.15) ...
Selecting previously unselected package linux-headers-5.4.0-1015-raspi.
Preparing to unpack .../245-linux-headers-5.4.0-1015-raspi_5.4.0-1015.15_arm64.deb ...
Unpacking linux-headers-5.4.0-1015-raspi (5.4.0-1015.15) ...
Selecting previously unselected package linux-headers-raspi.
Preparing to unpack .../246-linux-headers-raspi_5.4.0.1015.50_arm64.deb ...
Unpacking linux-headers-raspi (5.4.0.1015.50) ...
Selecting previously unselected package linux-modules-5.4.0-1015-raspi.
Preparing to unpack .../247-linux-modules-5.4.0-1015-raspi_5.4.0-1015.15_arm64.deb ...
Unpacking linux-modules-5.4.0-1015-raspi (5.4.0-1015.15) ...
Selecting previously unselected package linux-image-5.4.0-1015-raspi.
Preparing to unpack .../248-linux-image-5.4.0-1015-raspi_5.4.0-1015.15_arm64.deb ...
Unpacking linux-image-5.4.0-1015-raspi (5.4.0-1015.15) ...
Selecting previously unselected package linux-image-raspi.
Preparing to unpack .../249-linux-image-raspi_5.4.0.1015.50_arm64.deb ...
Unpacking linux-image-raspi (5.4.0.1015.50) ...
Selecting previously unselected package linux-raspi.
Preparing to unpack .../250-linux-raspi_5.4.0.1015.50_arm64.deb ...
Unpacking linux-raspi (5.4.0.1015.50) ...
Preparing to unpack .../251-linux-raspi2-hwe-18.04_5.4.0.1015.50_arm64.deb ...
Unpacking linux-raspi2-hwe-18.04 (5.4.0.1015.50) over (5.3.0.1030.20) ...
Preparing to unpack .../252-linux-headers-raspi2-hwe-18.04_5.4.0.1015.50_arm64.deb ...
Unpacking linux-headers-raspi2-hwe-18.04 (5.4.0.1015.50) over (5.3.0.1030.20) ...
Preparing to unpack .../253-linux-image-raspi2-hwe-18.04_5.4.0.1015.50_arm64.deb ...
Unpacking linux-image-raspi2-hwe-18.04 (5.4.0.1015.50) over (5.3.0.1030.20) ...

At the end of the upgrade process, do-release-upgrade calculates the obsolete packages and remove the following kernel packages:

  linux-headers-5.3.0-1017-raspi2
  linux-headers-5.3.0-1030-raspi2 linux-headers-5.4.0-1015-raspi
  linux-headers-raspi linux-headers-raspi2-hwe-18.04
  linux-image-5.3.0-1017-raspi2 linux-image-raspi2-hwe-18.04
  linux-modules-5.3.0-1017-raspi2 linux-raspi
  linux-raspi-headers-5.4.0-1015 linux-raspi2-5.3-headers-5.3.0-1017
  linux-raspi2-5.3-headers-5.3.0-1030 linux-raspi2-hwe-18.04

Please note that the flavour of the raspberry pi kernel changed from "raspi2" on 18.04 to "raspi" on 20.04. The transitional packages in 20.04 seem to be correct, so this behaviour is likely an issue with do-release-upgrade not handling this rename correctly and removing too many packages.

[Kleber Sacilotto de Souza (kleber-souza)]

I am attaching the screen log of the upgrade process.

[Brian Murray (brian-murray)]

@Kleber - one fix which you could test is adding "^linux-raspi$" to file removal_blacklist.cfg in the extracted dist-upgrader tarball. So run "do-release-upgrade -d" then cancel the upgrade then cd to "/tmp/ubuntu-release-upgrader-$TMPDIR". In that folder make your changes to removal_blacklist.cfg then run "sudo ./focal".

[Brian Murray (brian-murray)]

I ran a distribution upgrade with the changes found at https://pastebin.ubuntu.com/p/GxF2QHhrJT/ and both linux-headers-raspi and linux-raspi were kept. Here are portions of the main.log file:

2020-07-31 17:09:19,253 DEBUG package 'linux-headers-raspi' produces an unwanted removal 'linux-raspi', skipping
2020-07-31 17:09:21,085 DEBUG 'linux-headers-raspi' scheduled for remove but not safe to remove, skipping
2020-07-31 17:09:27,964 DEBUG skipping kernel metapackage 'linux-raspi'
2020-07-31 17:09:27,964 DEBUG 'linux-raspi' scheduled for remove but not safe to remove, skipping
2020-07-31 17:09:29,672 DEBUG Finish checking for obsolete pkgs
2020-07-31 17:09:29,747 DEBUG The following packages are marked for removal: acpid liblvm2cmd2.02 libpython3.6-minimal libisccfg160 ureadahead libargon2-0 acl linux-raspi-hwe-18.04 gcc-8-base libpython3.6-stdlib python3-asn1crypto libicu60 libirs160 linux-raspi-5.4-headers-5.4.0-1015 geoip-database perl-modules-5.26 libncurses5 python3-pam dnsmasq-base xdelta3 linux-headers-raspi-hwe-18.04 libpython3.6 python3.6 libisc169 ebtables libip6tc0 command-not-found-data libncursesw5 libprocps6 net-tools lxd libplymouth4 nplan liblxc1 libperl5.26 python3.6-minimal libiptc0 libisc-export169 libgeoip1 libtinfo5 linux-image-raspi-hwe-18.04 libnih1 liblvm2app2.2 libgdbm5 liblwres160 libntfs-3g88 mlocate libhogweed4 libreadline7 libdns-export1100 libnettle6 libevent-2.1-6 libidn11 multiarch-support btrfs-tools libffi6 dns-root-data libisccc160 libbind9-160 libjson-c3 libdns1100 liblxc-common libip4tc0 lxd-client uidmap libssl1.0.0 lxcfs

This looks right and I'll upload a fix today.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.10.8

---------------
ubuntu-release-upgrader (1:20.10.8) groovy; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: Ensure that the linux metapackage is
    never removed during a distribution upgrade. (LP: #1889449)

 -- Brian Murray <email address hidden> Fri, 31 Jul 2020 11:20:23 -0700

[Kleber Sacilotto de Souza (kleber-souza)]

@brian-murray,

I confirm that with your workaround from comment #3 the raspi kernels are not removed after the upgrade:

$ dpkg -l | grep "5\.4\.0\."
ii linux-headers-raspi 5.4.0.1015.50 arm64 Raspberry Pi Linux kernel headers
ii linux-image-raspi 5.4.0.1015.50 arm64 Raspberry Pi Linux kernel image
ii linux-raspi 5.4.0.1015.50 arm64 Complete Raspberry Pi Linux kernel and headers

[Łukasz Zemczak (sil2100)]

Hello Kleber, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

I tested this by downloading the new release upgrader tarball from (http://archive.ubuntu.com/ubuntu/dists/focal-proposed/main/dist-upgrader-all/current/focal.tar.gz) and extract it to /tmp/. I then ran 'sudo ./focal --frontend DistUpgradeViewText'. Here are the relevant portions of /var/log/dist-upgrade/main.log:

2020-08-03 22:30:38,866 INFO Using config files '['./DistUpgrade.cfg.bionic']'
2020-08-03 22:30:38,867 INFO uname information: 'Linux ubuntu 5.4.0-1015-raspi #15~18.04.1-Ubuntu SMP Fri Jul 10 09:20:51 UTC 2020 aarch64'
2020-08-03 22:30:39,629 INFO apt version: '1.6.12ubuntu0.1'
2020-08-03 22:30:39,629 INFO python version: '3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0]'
2020-08-03 22:30:39,637 INFO release-upgrader version '20.04.24' started
...
2020-08-03 23:47:33,025 DEBUG Start checking for obsolete pkgs
2020-08-03 23:47:34,240 DEBUG package 'linux-headers-raspi' produces an unwanted removal 'linux-raspi', skipping
2020-08-03 23:47:35,143 DEBUG 'linux-headers-raspi' scheduled for remove but not safe to remove, skipping
2020-08-03 23:47:41,473 DEBUG skipping kernel metapackage 'linux-raspi'
2020-08-03 23:47:41,474 DEBUG 'linux-raspi' scheduled for remove but not safe to remove, skipping
2020-08-03 23:47:49,878 DEBUG Finish checking for obsolete pkgs
2020-08-03 23:47:49,950 DEBUG The following packages are marked for removal: acpid liblvm2cmd2.02 libpython3.6-minimal libisccfg160 ureadahead libargon2-0 acl linux-raspi-hwe-18.04 gcc-8-base libpython3.6-stdlib python3-asn1crypto libicu60 libirs160 linux-raspi-5.4-headers-5.4.0-1015 geoip-database perl-modules-5.26 libncurses5 python3-pam dnsmasq-base xdelta3 linux-headers-raspi-hwe-18.04 libpython3.6 python3.6 libisc169 ebtables libip6tc0 command-not-found-data libncursesw5 libprocps6 net-tools lxd libplymouth4 nplan liblxc1 libperl5.26 python3.6-minimal libiptc0 libisc-export169 libgeoip1 libtinfo5 linux-image-raspi-hwe-18.04 libnih1 liblvm2app2.2 libgdbm5 liblwres160 libntfs-3g88 mlocate libhogweed4 libreadline7 libdns-export1100 libnettle6 libevent-2.1-6 libidn11 multiarch-support btrfs-tools libffi6 dns-root-data libisccc160 libbind9-160 libjson-c3 libdns1100 liblxc-common libip4tc0 lxd-client uidmap libssl1.0.0 lxcfs

So the new version of the release upgrader worked for me but I'd appreciate it if Kleber or somebody else would test it. Thanks!

[Kleber Sacilotto de Souza (kleber-souza)]

@brian-murray,

Running the new release upgrader tarball from the link you provided seems to have fixed the issue. After removing the unused package I still have the linux-raspi meta packages installed:

$ dpkg -l | grep linux | grep "5\.4\.0\."
ii linux-headers-raspi 5.4.0.1015.50 arm64 Raspberry Pi Linux kernel headers
ii linux-image-raspi 5.4.0.1015.50 arm64 Raspberry Pi Linux kernel image
ii linux-raspi 5.4.0.1015.50 arm64 Complete Raspberry Pi Linux kernel and headers

Thanks!

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.24

---------------
ubuntu-release-upgrader (1:20.04.24) focal; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: Ensure that the linux metapackage is
    never removed during a distribution upgrade. (LP: #1889449)

 -- Brian Murray <email address hidden> Mon, 03 Aug 2020 09:26:16 -0700