Stage "searching for obsolete software" takes a very long time (30 minutes)

[Jean-Baptiste Lallement]

[Impact]
Upgrades hang for a long time searching for obsolete software, frustrating users.

[Test plan]

1. With each, update, and proposed version do:

  0. launch a container / VM
  1. do-release-upgrade
  2. Note down the time it took for the obsolete software stage, and the packages marked for removal.

2. Check that the time is now significantly smaller than before

For example with the default ubuntu:jammy lxd container:

before: ~8s
2024-07-16 15:13:25,575 DEBUG Start checking for obsolete pkgs
2024-07-16 15:13:33,847 DEBUG Finish checking for obsolete pkgs

after: ~0.5s
2024-07-16 15:22:21,977 DEBUG Start checking for obsolete pkgs
2024-07-16 15:22:22,508 DEBUG Finish checking for obsolete pkgs

3. Compare the list of packages marked for removal: It should be the same or more packages.

before:
2024-07-16 15:13:33,907 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binuti
ls-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage

after:
2024-07-16 15:22:22,555 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binuti
ls-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage

This small container is not necessarily the best test case to demonstrate it and ensure correctness but the fastest one; it may also be suitable to run it in a desktop VM instead.

[Where problems could occur]
The upgrade process could crash in case the solver fails to solve the request it is being given. This is the last step of the upgrade though, so it is reasonable safe.

One could have skipped on a solver failure but this way we'd get crash reports
in the unlikely case they happen.

[Original bug report]

Test Case:
1. Fresh Bionic installation from 18.04.4 + all updates applied
2. Run: update-manager -d
3. Proceed with the upgrade.

Actual Result:
Everything goes well but the stage "Searching for obsolete software" takes a very long time.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: ubuntu-release-upgrader-core 1:20.04.18
ProcVersionSignature: Ubuntu 5.4.0-26.30-generic 5.4.30
Uname: Linux 5.4.0-26-generic x86_64
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
CrashDB: ubuntu
CurrentDesktop: ubuntu:GNOME
Date: Wed Apr 22 10:34:19 2020
InstallationDate: Installed on 2020-04-20 (2 days ago)
InstallationMedia: Ubuntu 18.04.4 LTS "Bionic Beaver" - Release amd64 (20200203.1)
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubuntu-release-upgrader
Symptom: ubuntu-release-upgrader
UpgradeStatus: Upgraded to focal on 2020-04-22 (0 days ago)
VarLogDistupgradeTermlog:

[Brian Murray (brian-murray)]

I restored this apt-clone file to an Ubuntu 18.04 LTS system and then upgraded to Ubuntu 20.04 LTS. The search for obsolete software only took a minute.

bdmurray@clean-bionic-amd64:~$ grep "obsolete pkgs" /var/log/dist-upgrade/main.log
2020-04-22 14:04:41,576 DEBUG Start checking for obsolete pkgs
2020-04-22 14:05:27,656 DEBUG Finish checking for obsolete pkgs

It's not immediately clear to me what is going on. This other cache operation of yours also took a long time.

2020-04-22 09:17:49,110 INFO cache.commit()
2020-04-22 09:42:51,188 DEBUG cache.commit() returned None
2020-04-22 09:42:51,332 DEBUG openCache()
2020-04-22 09:42:54,210 DEBUG /openCache(), new cache size 65224

[Brian Murray (brian-murray)]

I also did a spot check of open bugs in Launchpad for Mainlog.txt files with an extremely long times between starting and finishing the check for obsolete packages and I found none. Although, maybe I should check closed bugs too.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Fabricio G. Scaglioni (fgscaglioni)]

I was upgrading from version 18.04 to 20.04 when this bug has appeared.

2020-04-25 11:47:59,154 DEBUG Start checking for obsolete pkgs
2020-04-25 12:37:11,877 DEBUG Finish checking for obsolete pkgs

[RarSa (rarsa)]

I can confirm the problem. Including the delay in the cache.commit():

2020-04-26 13:05:59,023 INFO cache.commit()
2020-04-26 14:24:25,464 DEBUG cache.commit() returned None

For the obsolete packages
2020-04-26 14:32:27,989 DEBUG Start checking for obsolete pkgs
2020-04-26 14:53:12,608 DEBUG Finish checking for obsolete pkgs

Most of the packages report a "DEBUG package '<package name' produces an unwanted removal'gnome-software-plugin-snap', skipping

[piktogramm (piktogramm)]

Sorry, I can't add much, but the process on my machine (2Core 4T Intel Haswell, 16GB Ram, Samsung 860Evo SSD) took over 6 hours. Update was from 19.10 to 20.04.

$ grep "obsolete pkgs" /var/log/dist-upgrade/main.log
2020-04-30 20:17:50,550 DEBUG Start checking for obsolete pkgs
2020-05-01 02:39:51,833 DEBUG Finish checking for obsolete pkgs

[Dimitri John Ledkov (xnox)]

I wonder if something stray is holding locks or doing things, and thus we are waiting on locks to expire.

Do we like stop unatanded upgrades / apt-daily timer units when we start the ugprade? Cause they may have "cought" the lock between us finish update, and starting to check for obsolete pkgs.

I have managed to catch lock "half-way in between doing things" before.

[Lincoln Phipps (lincoln-phipps)]

I've got this right now and if I tail the /var/log/dist-upgrade/main.log I have...

DEBUG package 'foo' produces an unwanted removal 'libomp5-10', skipping
DEBUG 'foo' scheduled for removal but not safe to remove, skipping

Where 'foo' changes and takes solid core 100% 20 seconds cpu to move to next package. The libomp5-10 stays the same.

Upgrade is 19.10 to 20.04 using do-release-upgrade in a console.

[Steve Langasek (vorlon)]

Marking this incomplete since we don't have a reliable reproducer for it. If someone experiencing this problem can check for other processes holding a lock (Dimitri's question), that would be helpful.

[David Jung (djung)]

Same occurrence here form GUI 'updater'. Not a clean install - I upgraded from 17 first, then 18 LTS -> 20.04. See the same pattern in /var/log/dist-upgrade/main.log as seen by @Lincoln above. Timestamps span ~20mins.

[Jani Uusitalo (uusijani)]

Currently experiencing this. `lsof /var/lib/dpkg/lock` shows only "focal" as holding the lock. The same "focal" process is also eating up all of the CPU. No dpkg or apt processes are listed by `ps` either.

[John Rose (johnaaronrose)]

Same problem. What would happen if I stopped the upgrade (e.g. by cutting power on my Intel Barebones PC)?

[bsy (bennet-yee)]

37 minutes so far. Top shows final using 100% of a core. Top -H shows only one thread. Main.log shows the skipping messages.

[bsy (bennet-yee)]

Upgrading a second system. 47 minutes of CPU time burned so far.

[paul fox (pgf-launchpad)]

I've upgraded 4 systems from 18.04 to 20.04, and only one has shown this behavior. Will attach an strace of the focal process. I ran the strace for approximately 30 seconds.

[Noam Mor (noam-mor)]

Upgraded with do-release-upgrade, "searching for obsolete software" took about 50 minutes. Attaching upgrade log (partial, upgrade has not finished yet).

[Launchpad Janitor (janitor)]

[Expired for ubuntu-release-upgrader (Ubuntu Focal) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for ubuntu-release-upgrader (Ubuntu) because there has been no activity for 60 days.]

[Conrad Knight (iestynapmwg)]

A google search brought me here... Was this ever solved? I'm currently having the same problem, upgrading from 20.10 to 21.04. "Searching for obsolete software" was displayed about an hour and a half ago, and the line below says " eading state information... 52%" (I assume there's a display bug in the output of that last line and there should be a leading 'R').

One cpu is pegged at 100%, running the upgrade process. /var/log/dist-upgrade/apt.log is full of lines like:

Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) fonts-gujr:amd64 < 2:1.4 @ii K Ib >
Broken fonts-gujr:amd64 Depends on fonts-kalapi:amd64 < 1.0-4 @ii pR >
  Considering fonts-kalapi:amd64 10001 as a solution to fonts-gujr:amd64 0
  Removing fonts-gujr:amd64 rather than change fonts-kalapi:amd64
  MarkDelete fonts-gujr:amd64 < 2:1.4 @ii K Ib > FU=0
Done
  MarkDelete pmidi:amd64 < 1.7.1-1 @ii K > FU=1
  MarkDelete librsvg2-common:i386 < 2.50.3+dfsg-1 @ii K > FU=1
  MarkDelete fonts-navilu:amd64 < 1.2-3 @ii K > FU=1
Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) fonts-knda:amd64 < 2:1.3 @ii K Ib >
Broken fonts-knda:amd64 Depends on fonts-navilu:amd64 < 1.2-3 @ii pR >
  Considering fonts-navilu:amd64 10001 as a solution to fonts-knda:amd64 0
  Removing fonts-knda:amd64 rather than change fonts-navilu:amd64
  MarkDelete fonts-knda:amd64 < 2:1.3 @ii K Ib > FU=0
Done

main.log is filling with:

2021-04-24 01:44:16,280 DEBUG package 'python3.9-examples' produces an unwanted removal 'libsnack-alsa', skipping
2021-04-24 01:44:52,824 DEBUG 'python3.9-examples' scheduled for remove but not safe to remove, skipping
2021-04-24 01:44:52,969 DEBUG package 'firebird3.0-common-doc' produces an unwanted removal 'libsnack-alsa', skipping
2021-04-24 01:45:28,698 DEBUG 'firebird3.0-common-doc' scheduled for remove but not safe to remove, skipping
2021-04-24 01:45:29,665 DEBUG package 'libvdpau1:i386' produces an unwanted removal 'libsnack-alsa', skipping
2021-04-24 01:46:05,161 DEBUG 'libvdpau1:i386' scheduled for remove but not safe to remove, skipping
2021-04-24 01:46:05,313 DEBUG package 'python3-sss' produces an unwanted removal 'libsnack-alsa', skipping
2021-04-24 01:46:43,381 DEBUG 'python3-sss' scheduled for remove but not safe to remove, skipping
2021-04-24 01:46:43,517 DEBUG package 'libvo-amrwbenc0:i386' produces an unwanted removal 'libsnack-alsa', skipping
2021-04-24 01:47:19,690 DEBUG 'libvo-amrwbenc0:i386' scheduled for remove but not safe to remove, skipping
2021-04-24 01:47:19,830 DEBUG package 'ruby-svn' produces an unwanted removal 'libsnack-alsa', skipping

[Cruz Fernandez (cruz-fernandez)]

Similar to what @iestynapmwg said, while upgrading from Ubuntu 20.10 to 21.04 experiencing this slowness on step "searching for obsolete software". Mine took 36 minutes (in my Dell XPS 15 9550 that's not so old).

2021-05-14 08:17:28,617 DEBUG Start checking for obsolete pkgs
2021-05-14 08:41:34,610 DEBUG Finish checking for obsolete pkgs

Main difference is that my main culprit seems always package 'libqt5gui5'

2021-05-14 08:37:08,688 DEBUG 'libparams-validationcompiler-perl' scheduled for remove but not safe to remove, skipping
2021-05-14 08:37:08,776 DEBUG package 'libqt5concurrent5' produces an unwanted removal 'libqt5gui5', skipping

and lsof command returned "hirsute"

sudo lsof /var/lib/dpkg/lock
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1002/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1002/doc
      Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
hirsute 15215 root 16uW REG 259,5 0 1966936 /var/lib/dpkg/lock

[Gregory P Smith (gpshead)]

Judging by the 100% cpu utilization and entries in /var/log/dist-upgrade/apt.log... as an outside observer black box guess: is this thing loading the parsing the entire package dependency graph repeatedly once per potential package?

88% complete after 30 minutes of CPU time...

[Seán Ó Séaghdha (seanos)]

Happened for me on 2 machines today upgrading 18.04 -> 20.04.

[Shaya Potter (sjpotter)]

I don't understand how there can be a comment that its not reproducible. I upgraded from 18.04->20.04 today and hit it and then upgraded to 21.04 and hit it again harder.

i have no idea what its trying to do to remove obsolete software, but perhaps it has to be rethought or even made into an option if it can't be done efficiently (I have 129 obsoleete packages installed at the moment, can probably prune it down, but i'd rather do it by hand then have it burn a cpu for an hour+)

[Sukhamjot singh (sukhamjot2001)]

This has not been resolved
Just upgraded from Ubuntu 18.04 to Ubuntu 20.04
The total installation took around 3 hours
Removing obsolete softwares took around 1 hour.

I am using dual boot Intel i5 9th-gen 8gb Ram.

[Alan Porter (alan.porter)]

Typing this from my phone while I wait for my laptop to upgrade from 21.04 to 21.10. I did not record the time, but I've been waiting long enough to search for the problem and end up here.

[Ionut Andrei Oanca (andrei-oanca)]

Facing this as well. Upgrade from 20.04 to 21.04 went OK (minor issue with `openmpi-bin`). Currently upgrading from 21.04 to 21.10 and have been waiting on "Searching for obsolete software" for almost an hour now (still at 64%).

i5-3570K, 24GB of RAM, one core always at 100% usage. `/var/log/dist-upgrade/apt.log` shows lots of `pkgProblemResolver` going on.

[Johan Smolders (smojo1)]

Upgrading from 21.10 to 22.04 same issue. "Searching for obsolete software"

Burning 1 core for 100%

/var/log/dist-upgrade/apt.log is spammed with messages like

Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) linux-modules-extra-5.13.0-39-generic:amd64 < 5.13.0-39.44 @ii K Ib >
Broken linux-modules-extra-5.13.0-39-generic:amd64 Depends on crda:amd64 < 4.14+git20191112.9856751-1build1 @ii pR >
  Considering crda:amd64 10001 as a solution to linux-modules-extra-5.13.0-39-generic:amd64 -2
Broken linux-modules-extra-5.13.0-39-generic:amd64 Depends on wireless-crda:amd64 < none @un H >
  Or group remove for linux-modules-extra-5.13.0-39-generic:amd64
  MarkDelete linux-modules-extra-5.13.0-39-generic:amd64 < 5.13.0-39.44 @ii K Ib > FU=0

/var/log/dist-upgrade/main.log is spammed with messages like

2022-04-18 22:25:31,314 DEBUG 'fonts-vlgothic' scheduled for remove but not safe to remove, skipping
2022-04-18 22:25:31,377 DEBUG package 'libnspr4:i386' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:25:40,712 DEBUG 'libnspr4:i386' scheduled for remove but not safe to remove, skipping
2022-04-18 22:25:40,769 DEBUG package 'node-getpass' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:25:50,094 DEBUG 'node-getpass' scheduled for remove but not safe to remove, skipping
2022-04-18 22:25:50,152 DEBUG package 'libvpx6:i386' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:25:59,365 DEBUG 'libvpx6:i386' scheduled for remove but not safe to remove, skipping
2022-04-18 22:25:59,429 DEBUG package 'libmbedtls12' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:26:08,726 DEBUG 'libmbedtls12' scheduled for remove but not safe to remove, skipping
2022-04-18 22:26:08,778 DEBUG package 'libpango-perl' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:26:18,153 DEBUG 'libpango-perl' scheduled for remove but not safe to remove, skipping
2022-04-18 22:26:18,225 DEBUG package 'plasma-wallpapers-addons' produces an unwanted removal 'firefox-locale-en', skipping
2022-04-18 22:26:27,841 DEBUG 'plasma-wallpapers-addons' scheduled for remove but not safe to remove, skipping
2022-04-18 22:26:27,900 DEBUG package 'plasma-widgets-addons' produces an unwanted removal 'firefox-locale-en', skipping

I used the APT version of Firefox on 21.10, I read it's being replaced by the SNAP version? Might that have something to do with it?

[Davide Barasti (dbarasti)]

Upgrading from 22.04lts to 22.10 from a Thinkpad w/ AMD Ryzen™ 7 PRO 3700U

Hours in the Cleaning up phase (I 'd say after 4 hours the process is at 85%) with errors popping up every few seconds saying "Kinetic is not responding" -> Force quit or Wait.

Can I safely force quit the process? Is it going to damage the upgrade?

Really annoying situation

[Fabien C (fabich)]

Upgrading from 22.10 to 23.04 same issue:
 - "Searching for obsolete software" is at 83% after ~1 hour
 - Burning 1 core for 100%
 - A lot of logs in /var/log/dist-upgrade/main.log "XXX scheduled for remove but not safe to remove, skipping" or "package YYY produces an unwanted removal ZZZ, skipping"

[Julian Andres Klode (juliank)]

So at the moment this loops over all packages that seem obsolete, and tries to remove them and then undoes that if it removes other packages, using one actiongroup per package. This is horrible.

What we should do is use the ProblemResolver's resolve_by_keep something like this

with an actiongroup:
   problemsolver = apt_pkg.ProblemResolver(depcache)
   for package in obsolete:
       depcache.mark_delete(package)
   for package in not_obsolete:
       depcache.mark_keep(package)
       problemsolver.protect(package)
   problemsolver.resolve_by_keep()

Caveat is that this may also affect any other actions currently queued, but I think at this point there aren't any others.

[Gao Shichao (xgdgsc)]

Also happens when upgrading from 2204 to 2404

[Julian Andres Klode (juliank)]

https://code.launchpad.net/~juliank/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/465148 speeds this up by roughly 35x for the default lxc container. For larger upgrades, the speed up should probably be even faster.

[Simon Dalley (dalley-simon)]

Running 23.10 on a 6-core AMD Zen 3 CPU with 32GB RAM. Software Updater advertised availability of Noble Numbat 24.04 upgrade. All went OK until Cleaning up / "searching for obsolete software", 1 CPU core running "noble" went to 100%. Stayed like that for 4 hours. Eventually accepted the pop-up which offered to Force Close the app. Sounds like this bug, rather a showstopper for me. Fortunately there was a pre-upgrade snapshot that I could revert back to.

[Simon Dalley (dalley-simon)]

@juliank

[Simon Dalley (dalley-simon)]

https://launchpad.net/~juliank do you want me to test your patch? What would be the best way to go about this?

Since this bug doesn't always seem to bite everyone, it is possible that a pathological combination of installed packages might trigger it?

If you want, I can try and reproduce it with my particular setup.

[data (yavor-dobrev)]

I am also seeing this on two separate systems, while upgrading from 22.04 to 24.04. On the first system, with less packages installed the process took about 2 hours. On the second system, based on the current progress I estimate that it will take around 14 hours.

[DanglingPointer (ferncasado)]

I'm upgrading now from 22.04 to 24.04 and have copped this bug!

It's been running now for 2 hours!

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:24.10.5

---------------
ubuntu-release-upgrader (1:24.10.5) oracular; urgency=medium

  [ Nick Rosbrook ]
  * Only rewrite deb822 sources on upgrade
    - DistUpgradeController: migrate to deb822 before updating sources
    - DistUpgrade: don't make Deb822SourceEntry import conditional
    - DistUpgrade: only leave comment in sources.list if we migrated it
    - DistUpgrade: handle deb822 migration rollback on abort
    - DistUpgrade: clean up backup sources
  * Remove dead code
    - DistUpgradeQuirks: remove unused quirks
    - test: remove unused interactive_fetch_release_upgrader.py
    - DistUpgrade: remove old and unused DistUpgradePatcher
    - DistUpgrade: remove unused zz-update-grub
    - DistUpgrade: remove apt_clone.py symlink
    - DistUpgrade: remove imported/invoke-rc.d.diff
    - DistUpgradeController: remove .list update/re-write functions
    - DistUpgradeController: remove unused distro.py import and symlink
    - DistUpgrade: remove DistUpgradeQuirks.run
    - DistUpgrade: remove un-used PreRequists logic
    - all: remove unused config and test data
    - all: remove unused imports
    - DistUpgradeCache: remove unused methods and variables
  * DistUpgrade: fix several typos of 'upgrade'
  * DistUpgradeController: fix attribute typos
  * Run pre-build.sh: updating mirrors, demotions, and translations.

  [ Julian Andres Klode ]
  * Move the actiongroup out of the obsoletes loop
  * Reimplement obsolete removal with resolve_by_keep (LP: #1874272)

 -- Nick Rosbrook <email address hidden> Thu, 11 Jul 2024 15:04:57 -0400

[Bristow (web-frayssinet)]

This phase (migration from 2204 to 2404) took about 3 hours on my recent laptop (ThinkPad x13). Maybe because I have LaTeX installed?
I noticed that not all the processor's power was being used.
Only one core is running at 100%, and this changes regularly.

Perhaps some work on that front?

[Łukasz Zemczak (sil2100)]

Hello Jean-Baptiste, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Sami Kuisma (sku-8936)]

I am having the same problem. Stuck at "Searching for obsolete software", nearly an hour and now at 97%.
Upgrading 22.04 to 24.04 from command line.

CPU at 100% noble process. I have Ryzen 5950X + pcie 4 nvme samsung, so it is pretty fast but still taking almost an hour.

[Chris Halse Rogers (raof)]

An upload of ubuntu-release-upgrader to noble-proposed has been rejected from the upload queue for the following reason: "Changing the "The package %s is marked for removal..." string to an f"" string unnecessarily breaks translations, and it looks like the Amharic translation got broken (it retains the old translation, containing a '%s' - I'm not *sure* what python will do when it hits that, but it's not going to be what we want). Otherwise looks fine to me.".

[Łukasz Zemczak (sil2100)]

Hello Jean-Baptiste, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

I have verified the fix using ubuntu-release-upgrader from noble-propesed:

First, I upgraded a container without the fix:

nr@six:~$ lxc launch ubuntu:jammy j
nr@six:~$ lxc exec j bash
root@j:~# do-release-upgrade -d
Checking for a new Ubuntu release

= Welcome to Ubuntu 24.04 LTS 'Noble Numbat' =

The Ubuntu team is proud to announce Ubuntu 24.04 LTS 'Noble Numbat'.

To see what's new in this release, visit:
  https://wiki.ubuntu.com/NobleNumbat/ReleaseNotes

Ubuntu is a Linux distribution for your desktop or server, with a fast
and easy install, regular releases, a tight selection of excellent
applications installed by default, and almost any other software you
can imagine available through the network.

We hope you enjoy Ubuntu.

== Feedback and Helping ==

If you would like to help shape Ubuntu, take a look at the list of
ways you can participate at

  http://www.ubuntu.com/community/participate/

Your comments, bug reports, patches and suggestions will help ensure
that our next release is the best release of Ubuntu ever. If you feel
that you have found a bug please read:

  http://help.ubuntu.com/community/ReportingBugs

Then report bugs using apport in Ubuntu. For example:

  ubuntu-bug linux

will open a bug report in Launchpad regarding the linux package.

If you have a question, or if you think you may have found a bug but
aren't sure, first try asking on the #ubuntu or #ubuntu-bugs IRC
channels on Libera.Chat, on the Ubuntu Users mailing list, or on the
Ubuntu forums:

  http://help.ubuntu.com/community/InternetRelayChat
  http://lists.ubuntu.com/mailman/listinfo/ubuntu-users
  http://www.ubuntuforums.org/

== More Information ==

You can find out more about Ubuntu on our website, IRC channel and wiki.
If you're new to Ubuntu, please visit:

  http://www.ubuntu.com/

To sign up for future Ubuntu announcements, please subscribe to Ubuntu's
very low volume announcement list at:

  http://lists.ubuntu.com/mailman/listinfo/ubuntu-announce

Continue [yN] y
Get:1 Upgrade tool signature [833 B]
Get:2 Upgrade tool [1278 kB]
Fetched 1279 kB in 0s (0 B/s)
authenticate 'noble.tar.gz' against 'noble.tar.gz.gpg'
extracting 'noble.tar.gz'
[screen is terminating]
root@j:~# grep "checking for obsolete pkgs" /var/log/dist-upgrade/main.log
2024-08-12 19:48:42,704 DEBUG Start checking for obsolete pkgs
2024-08-12 19:48:51,918 DEBUG Finish checking for obsolete pkgs
root@j:~# grep "The following packages are marked for removal" /var/log/dist-upgrade/main.log
2024-08-12 19:48:51,974 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binutils-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblock...

[Nick Rosbrook (enr0n)]

bug 2076913 was just reported, which was introduced by this patch. It will be a quick follow up to fix, but this should not be released to -updates in the mean time.

[Łukasz Zemczak (sil2100)]

Hello Jean-Baptiste, or anyone else affected,

Accepted ubuntu-release-upgrader into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:24.04.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

Given the regressions related to this original patch, I re-did SRU verification using the latest upload:

nr@six:~$ lxc launch ubuntu:jammy j1
Launching j1
nr@six:~$ lxc launch ubuntu:jammy j2
Launching j2
nr@six:~$ lxc exec j1 bash
root@j1:~# do-release-upgrade --devel

[...SNIP...]

root@j1:~# grep "release-upgrader version" /var/log/dist-upgrade/main.log
2024-08-22 16:26:07,525 INFO release-upgrader version '24.04.16' started
root@j1:~# grep "checking for obsolete pkgs" /var/log/dist-upgrade/main.log
2024-08-22 16:33:00,543 DEBUG Start checking for obsolete pkgs
2024-08-22 16:33:14,349 DEBUG Finish checking for obsolete pkgs
root@j1:~# grep "The following packages are marked for removal" /var/log/dist-upgrade/main.log
2024-08-22 16:33:14,410 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binutils-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage

Then, in the other container:

nr@six:~$ lxc exec j2 bash
root@j2:~# do-release-upgrade --proposed

[...SNIP...]

root@j2:~# grep "release-upgrader version" /var/log/dist-upgrade/main.log
2024-08-22 16:26:48,604 INFO release-upgrader version '24.04.22' started
root@j2:~# grep "checking for obsolete pkgs" /var/log/dist-upgrade/main.log
2024-08-22 16:34:56,158 DEBUG Start checking for obsolete pkgs
2024-08-22 16:34:56,806 DEBUG Finish checking for obsolete pkgs
root@j2:~# grep "The following packages are marked for removal"
^C
root@j2:~# grep "The following packages are marked for removal" /var/log/dist-upgrade/main.log
2024-08-22 16:34:56,857 DEBUG The following packages are marked for removal: libperl5.34 python3.10 libctf-nobfd0 libblockdev-part2 libblockdev-swap2 libpcre3 libunistring2 libaio1 libmpdec3 libbinutils python3-jeepney libpython3.10-minimal libblockdev-fs2 libpython3.10-stdlib binutils-x86-64-linux-gnu irqbalance libbpf0 libtss2-mu0 libldap-2.5-0 isc-dhcp-common libisc-export1105 libappstream4 libpython3.10 libblockdev-loop2 libblockdev2 libblockdev-part-err2 isc-dhcp-client libctf0 libnsl2 libblockdev-utils2 python3-jaraco.classes libprocps8 binutils-common libdns-export1110 libsframe1 pkexec python3-keyring libgprofng0 libcbor0.8 python3.10-minimal perl-modules-5.34 ubuntu-advantage-tools gcc-12-base libblockdev-crypto2 libicu70 libnetplan0 policykit-1 binutils python3-secretstorage

The searching for obsoletes stage was much faster in the second container, and the set of removals is the same.

[Nick Rosbrook (enr0n)]

I have verified the fixes for the regressions caused by the initial upload of this fix, so I am going to remove the block-noble-proposed tag now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:24.04.22

---------------
ubuntu-release-upgrader (1:24.04.22) noble; urgency=medium

  [ Tobias Heider ]
  * Fix typo in force_obsoletes. (LP: #2076913)

  [ Nick Rosbrook ]
  * DistUpgrade: handle cache key error when removing obsolete packages
    (LP: #2077358)
  * deb2snap: switch snap-store and firmware-updater tracks
    (LP: #2077344)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

ubuntu-release-upgrader (1:24.04.21) noble; urgency=medium

  * DistUpgradeQuirks: disable stub resolver on upgrades from jammy
    (LP: #2043820)
  * DistUpgrade: show a message about denied package removals
    (LP: #2073278)
  * data: add cryptsetup-initramfs to removal_denylist.cfg
    (LP: #2067622)
  * DistUpgradeQuirks: abort if system is not /usr-merged
    (LP: #2058648)
  * DistUpgradeQuirks: disable cloud-init on upgrade
    (LP: #2075968)
  * DistUpgradeQuirks: abort if rabbitmq-server installed
    (LP: #2074309)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

ubuntu-release-upgrader (1:24.04.20) noble; urgency=medium

  [ Mate Kukri ]
  * DistUpgradeQuirks: make sure GRUB "cloud_style_installation" is set in cloud images (LP: #2054103)

  [ Julian Andres Klode ]
  * Move the actiongroup out of the obsoletes loop
  * Reimplement obsolete removal with resolve_by_keep (LP: #1874272)

  [ Nick Rosbrook ]
  * Run pre-build.sh: updating mirrors, demotions, and translations.

 -- Nick Rosbrook <email address hidden> Wed, 21 Aug 2024 12:55:15 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.