Apt deletes ubuntu-desktop during dist-upgrade

[Alexander Bakker]

[Impact]
gnome-shell gets removed on upgrades of gnome-shell and mutter if due to phasing we can only upgrade src:gnome-shell.

[Test plan]
This adds a minimal test case to the test suite that reproduced the issue and verifies the fix, test plan consists of the autopkgtests.

[Where problems could occur]
We could see a resurgence of bugs like LP#1990586 where the solver failed to run at all because things got too complex, however this is a bit more unlikely as we now use the by-keep resolver to handle rolling back phased updates.

We also saw an issue with a phased update's dependencies being installed despite the phased update being hold back. We saw that both before in `apt upgrade` and with the fix for this bug, we also introduced it to `dist-upgrade`, but this is also fixed and tested in these uploads, basically we skip marking for install (which in turn caused me to discover we need to check a different version).

[Other Info]
gnome-shell Depends: gnome-shell-common (= ${binary-Version), mutter (>= matching)

So when mutter cannot be updated due to phasing, gnome-shell becomes non-installable, but gnome-shell-common can be updated, so APT decided to remove gnome-shell and the meta packages in its infinite wisdom.

The fix addresses this by resolving the dist-upgrade as if there were no phasing, and then retroactively marks phases for keep and then anything broken by that for keep.

This required some restructuring because normally we'd also keep broken Recommends back, but here dist-upgrade may have decided that was ok, so we need to build an allowlist of where Recommends can be broken to avoid undoing unrelated changes.

[Original bug report]

This morning I got surprised by my laptop booting to a tty instead of a desktop environment. It turned out that the entire desktop environment was no longer present on my machine. Doing an apt install ubuntu-desktop-minimal resolved the issue.

The machine had been running for a while. Looking at the apt logs, it looks like apt deleted ubuntu-desktop on its own during a dist-upgrade a couple of weeks back.

Start-Date: 2023-06-08 14:20:46
Commandline: apt dist-upgrade
Requested-By: alex (1000)
Upgrade: gnome-shell-common:amd64 (44.0-2ubuntu3, 44.1-0ubuntu1)
Remove: gnome-shell:amd64 (44.0-2ubuntu3), ubuntu-desktop:amd64 (1.501), gdm3:amd64 (44.0-1ubuntu1), gnome-shell-extension-desktop-icons-ng:amd64 (46+really47.0.2-3), gnome-shell-extension-appindicator:amd64 (53-1), ubuntu-session:amd64 (44.0-1ubuntu1), gnome-shell-extension-manager:amd64 (0.4.0-1), gnome-shell-extension-ubuntu-dock:amd64 (79ubuntu2.23.04.1), ubuntu-desktop-minimal:amd64 (1.501)
End-Date: 2023-06-08 14:20:48

I'm using the following version of Ubuntu:
Distributor ID: Ubuntu
Description: Ubuntu 23.04
Release: 23.04
Codename: lunar

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/2025462/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Alexander Bakker (realalexbakker)]

The reason I filed the report against Ubuntu in general is that I don't actually know which package is the culprit here. Is it apt? Is it gnome-shell-common? Is it ubuntu-desktop-minimal? Is it any of the other packages that were automatically removed?

[Brian Murray (brian-murray)]

I'm moving this to apt so the maintainer can have a look at it.

[Julian Andres Klode (juliank)]

Please attach /var/log/apt/history.log files showing at least the upgrade and the previous one.

[Julian Andres Klode (juliank)]

To debug the solver we'd also need an EDSP dump from that operation but I think we're out of luck.

[Julian Andres Klode (juliank)]

I believe this is a regression from https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1990586 which I know sounds hilarious because it was fixing another issue like this but here is what I believe happened:

- mutter and gnome-shell were released on the same day
- gnome-shell was eligible to be upgraded on the machine, mutter was still phasing
- with the change, we marked mutter for keep back
- the upgrade resolution ran, it was not able to upgrade gnome-shell anymore because it depends on the new mutter.

Before fixing the other bug, phasing had no effect on the resolution of the upgrade, and then we reverted the upgrade paths depending on the phasing, which resulted in no removals due to phasing. But the upgrade problems became to complex, breaking the solver.

[Julian Andres Klode (juliank)]

Written a minimal test case reproducing the issue, and wrote a fix in

https://salsa.debian.org/apt-team/apt/-/merge_requests/299

[Julian Andres Klode (juliank)]

The regression fix causes an issue whereby any new dependencies or recommends of updates that are phasing will be installed, we're currently investigating how to solve this, I believe this is also a problem in the apt upgrade code path (which can install new packages), but not in apt-get upgrade.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.7.2

---------------
apt (2.7.2) unstable; urgency=medium

  [ A. Maitland Bottoms ]
  * Do not fail on systems running in FIPSmode.

  [ Julian Andres Klode ]
  * Upload to unstable
  * update: Add notice about missing Signed-By in deb822 sources
  * doc: Bump distro versions for apt-key removal
  * dist-upgrade: Revert phased updates using keeps only (LP: #2025462)
  * Do not mark updates for install that are still phasing
  * Fix snapshot crashes with multiple components inside Ubuntu docker
    image

 -- Julian Andres Klode <email address hidden> Wed, 12 Jul 2023 17:23:26 +0200

[Andreas Hasenack (ahasenack)]

I checked previous SRUs, and we do rely on the automated tests for such changes, as it can be difficult to fabricate something "in real life" that would remain valid for the time we need it to be. I'm glad tests are being added.

This being a native package, and imported into the archive without rich history, it's a bit hard to follow what was described in d/changelog and the actual changes for a package I'm unfamiliar with, but I checked with the upstream branch in salsa, and it looks ok.

The po/* changes, and doc changes, are just about the version, so should be no issue for translators.

The same set of changes is applied to jammy and lunar, and in devel.

[Andreas Hasenack (ahasenack)]

Hello Alexander, or anyone else affected,

Accepted apt into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.6.0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Hello Alexander, or anyone else affected,

Accepted apt into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt/2.4.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted apt (2.6.0ubuntu0.1) for lunar have finished running.
The following regressions have been reported in tests triggered by the package:

auto-apt-proxy/14 (amd64)
devscripts/2.23.4ubuntu1 (amd64, arm64, armhf, ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/lunar/update_excuses.html#apt

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[reisig (reisig)]

Please check this related bug report with 11 affected users on Ubuntu Jammy:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2030262

[reisig (reisig)]

I think, to properly test the patch, we need some phasing test-packages that show the bug (removals) with the old apt. To then test the situation with the patched apt.

[Julian Andres Klode (juliank)]

The integration tests on autopkgtest.ubuntu.com have passed for the apt/2.4.10 and apt/2.6.0ubuntu0.1 uploads.

[Guy Rouillier (guy-rouillier)]

I was affected by this issue on Ubuntu MATE 22.04. I wanted to note that I was doing a routine upgrade through Update Manager, and *not* a dist-upgrade as mentioned numerous times in this and linked issues. I see in this issue the focus changed from dist-upgrade to apt, so I believe my case is covered now. I'm posting to reaffirm that the problem is much broader than just dist-upgrade.

Thank you.

[Julian Andres Klode (juliank)]

No the issue is very much specific to that dist-upgrade code path and no other code path in apt.

Yes, Update-Manager calls the dist upgrade code so it may or may not be affected as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.6.0ubuntu0.1

---------------
apt (2.6.0ubuntu0.1) lunar; urgency=medium

  * dist-upgrade: Revert phased updates using keeps only (LP: #2025462)
  * Do not mark updates for install that are still phasing
  * Compare SHA256 to check if versions are really the same (Closes: #931175)
    (LP: #2029268)
  * Branch CI and metadata for lunar, do not require test user to have UID
    1000, the lunar image has added an ubuntu user with that uid.

 -- Julian Andres Klode <email address hidden> Wed, 02 Aug 2023 16:05:30 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for apt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.4.10

---------------
apt (2.4.10) jammy; urgency=medium

  * dist-upgrade: Revert phased updates using keeps only (LP: #2025462)
  * Do not mark updates for install that are still phasing (same bug)
  * Compare SHA256 to check if versions are really the same (Closes: #931175)
    (LP: #2029268)

 -- Julian Andres Klode <email address hidden> Wed, 02 Aug 2023 15:15:58 +0200