When apt keeps back packages due to phased updates, it should list them separately

[Sergio Callegari]

After phased updates have been introduced, it may happen that apt upgrade shows packages as upgradable but ends up not upgrading them. In this case the packages are indicated as being "kept back".

Unfortunately, the feedback provided about this to the user is not very informative. The user sees the packages being kept back and thinks something is going wrong on the system.

When packages are kept back because of phased updates, apt should say so e.g., it should say that the upgrade is delayed.

Incidentally note that aptitude does not respect phased updates.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: apt 2.4.7
ProcVersionSignature: Ubuntu 5.15.0-47.51-generic 5.15.46
Uname: Linux 5.15.0-47-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: KDE
Date: Tue Sep 6 10:05:14 2022
EcryptfsInUse: Yes
InstallationDate: Installed on 2020-02-16 (933 days ago)
InstallationMedia: Kubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
SourcePackage: apt
UpgradeStatus: Upgraded to jammy on 2022-06-03 (94 days ago)

[Seth Arnold (seth-arnold)]

I have seen many people on IRC *very* upset after wasting a lot of time trying to install updates that apt will not let them install. Fixing this is critical to our reputation.

Thanks

[Julian Andres Klode (juliank)]

We can't really store reasons for why something is kept back. And if we managed to do that, we'd end up with an untranslated line in the middle of the output for every non-English locale until it's translated and apt updated (it does not use language packs).

[Brian Foster (blfoster)]

Yeah, I was confused by this at first too. Here's a hint, try "apt-mark showhold":

   $ apt-mark showhold
   $

If nothing is printed (as above), then the "issue" PRESUMABLY isn't local to your system; ergo, the cause is PROBABLY at the server, such as a phased update. Relax. It will probably sort itself out in a few days. (This hint is from https://askubuntu.com/questions/640986/how-to-get-a-list-of-installed-packages-held-back-from-upgrade "How to get a list of installed packages held back from upgrade?" (APT, etc., experts can please correct).)

As per above (#3), whilst the reason isn't stored, that does not mean apt(8)'s message cannot be improved. For example, instead of the current:

   The following packages have been kept back:
      PKG...

better might be something like:

   The following packages have been kept back:
      PKG...
   Some or all of those packages may be deliberately "on hold" on the server;
   please see manual page XXX(N) for information.

with (obviously) appropriate text in the manual page. (Exactly which man page is left to the experts.)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Seth Arnold (seth-arnold)]

apt-cache policy knows when packages are phased; when apt needs to report that packages are held back, apt could look up each one to report phased status.

Not fixing this because the strings need translating is an argument for declaring APT a finished project and moving on to the Next Big Thing.

[Robie Basak (racb)]

> We can't really store reasons for why something is kept back.

How about:

When something is scored down because it is not phased, make a note. When the scoring is all done, take the list of held back packages, and identify all those that have a note against them. Then print (up to) two lists:

The following packages are being phased and have been kept back:
   ...

The following [additional] packages have been kept back:
   ...

(use two translatable strings for the second message, adding "additional" if the first list was non-zero).

Of course the exact strings can be adjusted, but I hope you see my point in fixing the "we don't know what final reason is because the different scoring adjustments get conflated" problem.

> And if we managed to do that, we'd end up with an untranslated line in the middle of the output for every non-English locale until it's translated and apt updated (it does not use language packs).

We should certainly be able to fix this in the development release and get translations in the usual way. For SRU, we could wait for translations, or not, but no need to block on a fix in the development release for this.

[Seth Arnold (seth-arnold)]

An alternative, proposed by user avih on IRC, is to not report any of these packages to the user at all:

<avih> however, these phased updates are quite a big list which adds a lot of noise to my regular dist-upgrade, and it interferes with me reviewing what's about to get updated
<avih> the kept back list is quite bigger than the list of things to update...
<avih> arraybolt3: if this is indeed the standard order of things, why am i being shown at all what it's NOT going to install for reasons not related to errors or conflicts?

I can see a lot of appeal to not telling the user information -- from their perspective, the packages don't actually exist yet.

Maybe it'll cause confusion if of two machines sitting right next to each other, one can see the updates and the other cannot. That's not ideal.

But holding information back from the user doesn't require new strings, and casual users with one machine might never notice.

It's just fun to see an alternative idea that's 180 degrees different from my initial thought. :)

[Julian Andres Klode (juliank)]

I want to say I agree with avih and had the same idea but I had no chance to update the bug yet. Currently sorting out actual upgrade failures that might be regressions from the new phasing though.

It remains to be seen if we can do this safely. We need to add a new flag "PhasedKeep" or something to the depcache and then make sure to always clear this if we do not keep it.

[mkoniecz (matkoniecz)]

How can I simply disable phased packages? (I was hit numerous times by outdated packages, never had problem with update causing problems so for me this just makes things worse anyway)

[Sergio Callegari (callegar)]

As the original reporter, I am a bit unconvinced by this change of perspective in how to fix the issue. I agree that being silent is easier, but does not feel quite the right thing to do for me, at least unless an option or something is added to see which upgrades are in progress and phased back.

- It is untrue that from the user perspective these packages do not exist yet, as suggested.

> from their perspective, the packages don't actually exist yet.

The packages do exist and can be quite visible to the user. If you `apt install`, it is the most recent version of the packages that gets installed.

- It is generally considered bad practice to open bugs without first trying the most recent version of some troublesome software

Hiding the phased out packages takes away the opportunity to users who are experiencing problems to try the most recent version before opening a bug.

- When neighboring machines start running different versions of the software and the user has the belief that they should be aligned because they use the same distribution, if trouble occurs the user may be lead to wrong conclusions (e.g. inculpate the hardware).

Again the user should be able to easily check if there is the opportunity to try the last version of something when that something is giving trouble.

- Finally, I am afraid that being non-transparent may lead users to develop non standard hacks to see (and maybe immediately install) the phased out packages. Incidentally, currently `aptitude upgrade` installs the phased out packages.

[Julian Andres Klode (juliank)]

If phasing upgrades are hidden in upgrade output then the output shows you relevant info to the update only which is nice. I kind of feel like `apt upgrade` and friends should show you only unexpected things. Like often people install 100 packages and then miss that they also remove 20 that they did not want to.

Phased updates are list listed as upgradable in `apt list --upgradable`, in the `?upgradable` pattern, and can be updated using the `install` command (e.g. `apt install a-phased-package`); the policy command tells you if it is indeed phasing or not, but not whether it's phasing for you.

I admit it's confusing, but the next iteration should be to hide those upgrades (as they were for months) and then we can iterate on a nicer interface for the other side cases.

[Sergio Callegari (callegar)]

Would it then be possible to:
1. silence `apt upgrade` wrt the kept back packages
2. update `apt list --upgradable` to say which upgradable packages are phased out (i.e. not upgraded right now by `apt upgrade`
3. Update the bug report guidelines suggesting to `apt install` the troublesome package and retrying before opening a bug
4. update `aptitude` to behave as `apt` wrt to kept back packages?

[Benjamin Schmid (benbuntu)]

Even as a decade-old Ubuntu user this "The following packages have been kept back:" was a UX nightmare for me. It took me several runs of article reading, many unsuccessful "apt-mark showhold", breaking "apt install X" runs and still no clue at all about the background on these errors suddenly reappearing on 22.04 systems. I was this days old when I eventually was able to realize that this is all about "phased upgrades".

The implementation or effect from a UX is really utterly opaque. I'd go as far as to suggest: Refrain from phased upgrades until Ubuntu is able to signal apt users the reasons for packages held back!

In my decade-old history, "packages kept back" were 100% always some really, really bad borked stuff, i.e., on dist-upgrades. Always something to take immediate care about to not threat system integrity or automatic updates and hence threatening hence the system security. A reason why it never occurred to me to just ignore those line.

[Gone fishing (humpmihk-msn)]

I've been using Ubuntu since Warty or Breezy, and I have had very few incidents of an update breaking the system. I think an update in Hardy broke the X server once. Getting to the point - this phased update system had me thinking something had gone horribly wrong with apt and the package database. Maybe the cure is not worse than the problem, but it appears to be.

I also "fixed" the problem by manually installing the held-back packages, which I can now see was a bad idea.

I would suggest that, as we now have apt messages:

# News about significant security updates, features and services will
# appear here to raise awareness and perhaps tease /r/Linux ;)
# Use 'pro config set apt_news=false' to hide this and future APT news.

Maybe added to the news could be something about phased updates such as:
# You may see packages being held back as part of phased updates - these packages are being held back to ensure system stability.
# If you want to turn off phased updates, create a file in /etc/apt/apt.conf.d, containing the following lines:

Update-Manager::Always-Include-Phased-Updates;
APT::Get::Always-Include-Phased-Updates

[Seth Arnold (seth-arnold)]

So far I've been arguing that apt should be more verbose about phasing, and why these packages are held back. A friend has suggested that instead apt should say *nothing*. I can see the appeal.

Thanks

[Adrien Nader (adrien)]

The issue with being less verbose is that users will end up with the same issue when two neighbor machines have different updates. This also applies to machines belonging to different people as soon as these people discuss about a but that could be caused or solved by these updates.

I'd prefer to see a new line group which lists updates "rolling out over the next few days". It seems to be the same overall complexity to implement and it would be much much more informative. Hiding again seems like a step too far back to me.

[Jeffrey Walton (noloader)]

Yes, please fix.

From a security standpoint, we can't tell the "good" held backs (Phased Updates) from the "bad" held backs (problem with package). So we assume its a package problem, go in with a hammer, and use Aptitude's safe-upgrade to force the updates. Now we've subverted/undermined the Phased Updates when there was no problem.

[Davison Long (number-one)]

I for one don't like the idea of hiding the phased update info unless there is also a flag to show it. Not showing the info at all just obscures the issue even more and would lead to a lot of unnecessary confusion when odd situations might occur.

I also think it would be far better to have a separate non-translated message for packages held back due to phased updates and then add translations later. The current confusion is occurring world-wide; an English message about in process phased updates is going to provide a lot more useful info to a non-English speaker than what is happening now. Like others have mentioned, I also am a long-time Debian and Ubuntu user and this issue took WAY longer than it should have for me to understand what was actually going on since there was absolutely NO notices or info about phased updates at all in any apt commands or config files (except for apt policy on a specific package).

[Walter (wdoekes)]

Haha, yea, long time debian/ubuntu user here too. The only way I found out and know was because my script that parses apt-policy failed [1]. Pure luck [2].

Also seconding the do-translations-later. I don't know if there are statistics available for this, but based on anecdotal evidence alone, I say that many people run their OS in English anyway. And that the average debian/ubuntu user is used to more English than average users of proprietary operating systems..

[1] https://github.com/ossobv/vcutil/commit/9a34b973491f7165790f8c239f1b1c39bf4d1687
[2] Yay for defensive coding.

[Mehmet Can Cömert (cancomert)]

My current user experience is like following:
I just updated kept-back packages by mistake today by using apt install command.
I thought it was due to a wrong mark of Manuel tag in apt and marked them with apt-mark auto again.
Then after a bit search in internet I realized that it is due to a phased update policy, which makes sense.

I think either message should not shown at all (as suggested above)
or if you want to show the list, please state somehow that this is only info and we do not have to do something about it, such as:
The following packages have been kept back as a part of phased update policy of Ubuntu:

Just stating packages are kept back, gives the impression that something is broken in the dependency resolution.

[lochfynne (lochfynne)]

I totally agree with the previous writer.
Until reading this article, I manually upgraded Kept-back packages.

I support the idea of phased upgrades, but please create a separate notification stating something like:
The following packages have been kept back as a part of phased upgrade policy of Ubuntu: <list of packages affected>

This allows the user to deal with any other package issues separately.

[Lin Manfu (linmanfu)]

I was also confused by this change in behaviour, even though I knew about Phased Updates. I don't often update with apt now and had just forgotten about them.

I support the solution of listing phased updates separately.

I do not think the translation issue should hold this back. If apt has not internationalisation capability, then presumably other apt text such as "The following packages were automatically installed and are no longer required:" is not translated. Nor is the "News about significant security updates..." message is not translated either (and that contains a joke about Reddit, which is even more likely to confuse non-English speakers). And many non-English speakers will be very familiar with the process of copying a message they don't fully understand into a search engine to get more information and/or translation (maybe not an option for server users but we can expected them to use man apt).

[Sergio Callegari (callegar)]

There is a further problem, I do not know if it should be a separate issue.

The updates icon in the system tray now appears also when there are in fact no updates that can be applied. Don't know if this actually depends on the presence of updates that are phased out or on the presence of Ubuntu Pro (esm-apps) that are not enabled.

[Sergio Callegari (callegar)]

After further investigation, I would say that "Discover" wants to upgrade a package (libmm-glib0) notwithstanding the fact that it is phased and that apt does not want to upgrade it.

Funny enough, Discover does not indicate the other phased packages (it is up to 24!) as upgradable.

[Nathan Collins (ntc2)]

My vote is to remove this "phased updates" feature until the UI can be fixed. Right now I run `apt update` and it tells me "N packages can be upgraded", then I run `apt upgrade` and it tells me no packages will be upgraded because "N packages are held back". There is no mention of "phased updates". Very confused, wasting my time on this after upgrading to 22.04. Have been using Ubuntu since 2005 and found this very confusing.

[Xiao Wan (xwcal)]

To add a data point and save others from preventable headaches, I encountered a related bug and reported it here:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2017399

If anyone cares, here are my two cents regarding phased updates:
https://github.com/xwcal/ubuntu-apt-bug#a-short-critique-of-phased-updates

Or if lack of markdown formatting is not an issue, here is goes:

# A Short Critique of Phased Updates

Some thoughts after I stumbled on phased updates during a recent migration from Bionic to Jammy:

## Randomness and Repeatability

Per [Debian repository specs](https://wiki.debian.org/DebianRepository/Format#Phased-Update-Percentage), randomization is achieved by the following means:

```
To determine whether an update is applicable to a given machine, a client shall seed a random number generator (APT uses std::minstd_rand) with the string:

 sourcePackage-version-machineID

where sourcePackage is the name of the source package, version the version, and machineID the contents of /etc/machine-id.

It shall then extract an integer from a uniform integer distribution between 0 and 100.
```

To the uninitiated: this is not how a random number generator is meant to be used. If you want a repeatable sequence of random numbers, you seed the RNG once and then draw the desired number of samples. You don't reseed the RNG for each new sample, since if your seeds are already random, then the RNG step is pointless, whereas if your seeds aren't random, say, in an extreme case, if you always seed with the same number, your "random" number is guaranteed to be the same each time.

To achieve repeatable randomness with respect to both the package and machine, there is an obviously superior choice in terms of both randomness and portability: **just use a cryptographic hash like sha256**.

This begs the question of how repeatable the design really is.
To quote [https://wiki.debian.org/MachineId](https://wiki.debian.org/MachineId):

```
what is the machine id actually used for?

    a comprehensive list is probably not possible without grepping the code
```

That's to say, although for repeatable experiments it's foreseeably necessary to mess with the machine ID or share it publicly,
the consequences of such tinkering or disclosure are hard to assess, since nobody knows for sure how and where the machine ID is and will be used.

This leads to an easily conceivable alternative: since apt only uses the machine ID for the purpose of repeatable randomization, how about designating another user configurable value stored in, say `/etc/apt/phased-update-random-seed`, only for this particular purpose and by default initialized to the machine ID?

## User Choice

From [this response](https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2017399/comments/1) to my bug report (one question I forgot to ask is why phase security updates at all) and a comment on [askubuntu](https://askubuntu.com/questions/1431940/what-are-phased-updates-and-why-does-ubuntu-use-them) that doesn't cite any source, it does appear that consideration has been given regarding the need to treat different packages (at least different classes of packages) differently when it comes to phasi...

[Julian Andres Klode (juliank)]

I did not make any changes to the design when moving it from Update-Manager in apt, it's been this way for 10 years now, it's a proven concept.

The initial implementation used and md5 but the math gets complex when dealing with 128 bit integers.

You can easily override the machine-id for apt by setting the apt.conf option, /etc/machine-id is a default. It's general consensus that everything should be using the same machine id though.

[Xiao Wan (xwcal)]

@juliank Thanks for the tip about machine id. May I also ask, for repeatability, is there an official/supported way to run (unit) tests on the upgrade logic alone, say, by feeding `/var/lib/dpkg/status`, index files and maybe some environment variables/APT::* preferences to `apt-get` and then running a simulation?

As you have seen in https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2017399, the least amount of state for a reproducible report involved a livecd iso. And if I had been much further down the road it would have been impossible to trace the issue to the "original" state that everybody else also has access to.

Regarding md5, the first or last 64 bits (or 32bits for i386) already give a good random number for this purpose. There is no need to deal with 128bit integers.

[Julian Andres Klode (juliank)]

Generally for reproducing issues I set Phased-Update-Percentage to 0 on the packages the user had phased so it's independent of machine-id, it's not been a problem in my experience to reproduce issues.

[Julian Andres Klode (juliank)]

One thing I just noticed is that if you have something like gnome-shell and mutter where gnome-shell depends on the new mutter and only mutter is "not for us" we end up with something like:

The following packages have been kept back:
  gnome-shell gnome-shell-common mutter

with phasing specific message this would look like this:

The following packages have been kept back as they are phasing:
  mutter
The following packages have been kept back:
  gnome-shell gnome-shell-common

Because we kept back mutter and it is phasing, but the gnome-shell update is not anymore, it is kept back because it has broken dependencies on the mutter that is phasing.

[Roger Tawa (rogertawa)]

Julian, if apt knows that mutter is being held back due to phasing, and that gnome-shell and gnome-shell-common are being held back because of the mutter dependency, then it should be possible for the message to look something like:

  The following packages have been kept back as they are phasing or depend
  on a phased package:
    gnome-shell gnome-shell-common mutter

[Julian Andres Klode (juliank)]

It doesn't know that, no. It can guess that a phased package was kept back due to phasing, but the solvers don't track reasons for any decisions so it can't go past that.

[Julian Andres Klode (juliank)]

Proposed fix in:

https://salsa.debian.org/apt-team/apt/-/merge_requests/327/diffs?commit_id=483e3b5c49762306c0a9f54117fd70cff43af4be

The corner case with kept back packages not due to phasing ends up with a notice before the prompt:

Reading package lists...
Building dependency tree...
Calculating upgrade...
The following NEW packages will be installed:
  phased-new
The following updates have been deferred due to phasing:
  phased phased-dep phased-depends-ready-dep
The following packages have been kept back:
  depends-phased-dep ready-dep
The following packages will be upgraded:
  depends-phased-new phased-security
2 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
Need to get 0 B/126 B of archives.
After this operation, 43.0 kB of additional disk space will be used.
N: Some packages may have been kept back due to phasing.
Do you want to continue? [Y/n]

There may be a nicer way to do this.

[Julian Andres Klode (juliank)]

2.7.11 is in proposed now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apt - 2.7.11

---------------
apt (2.7.11) unstable; urgency=medium

  [ David Kalnischkies ]
  * Remove erroneous -a flag from apt-get synopsis in manpage
  * Support -a for setting host architecture in apt-get source -b

  [ Julian Andres Klode ]
  * For phasing, check if current version is a security update, not just previous ones
    (LP: #2051181)
  * Add public phased update API
  * Add a new ?phasing pattern
  * Add the ?security pattern
  * Show a separate list of upgrades deferred due to phasing (LP: #1988819)

 -- Julian Andres Klode <email address hidden> Tue, 13 Feb 2024 16:31:00 +0100

[Geekley (geekley)]

In the proposed fix, will there be some sort of flag and/or configuration for apt and apt-get to let me NOT see this list of "deferred due to phasing" packages? I agree it's important to list them for users who want to see it, but it's also important to not bloat the output with a bunch of extra info most users should not (or won't want to) worry about. Often this list of packages is quite long and distracting.

There should be something like APT::Get::Show-Deferred = --show-deferred and --no-show-deferred.

IMHO, not listing should be the default. I think most users would not need such verbosity, and by default you could simply show a message like "Some updates are deferred for you due to phasing" if this list would have been non-empty.
When users see this message, they can always use --show-deferred if they need the list; this would change the message to "The following updates are deferred for you due to phasing:" (list of packages).
If they use --no-show-deferred, then not even this message would appear.

[Julian Andres Klode (juliank)]

Geekley I personally agree and would go a lot further and hide even most dependencies (you don't really care which libraries you are installing, just about choices made, e.g. if there's an a | b dependency it should tell you that it picked a).

So if you want to think about it that terse mode would end up looking something like:

    Installing 5 specified packages, 10 upgrades and 30 new dependencies:
        - Choosing banana to satisfy foo Depends: banana | apple
    Removing 30 packages:
        - package1
        ...

At the moment there is no option in between full output and no output, though, and there is opposition to adding more output modes upstream.

[David Kalnischkies (donkult)]

Not sure who all the upstream(s) involved might be, but from my personal PoV at least you can add all the options you like… the topic gets harder if we talk defaults & changing (e.g.) the lists completely (like that tabular verbose-explosion thingy from apk or whatever it was). At some point it might make sense to extended apt-patterns so that current (and future) lists can be expressed in them and then add some more options to format those lists/tables/… at which point we could have different templates and so options/choices galore. I think aptitude has formatting to some extend of its lists. One of my first apt patches that was never merged was actually about reordering/coloring the lists… that failed, so I am very positive that a much bigger yak will be shaved more easily and faster many years later. ;)

Precedence of the initial ask is 'can be autoremoved' btw, which is not displayed, displays a full list, an even fuller list in version mode or displays a single line with how many packages could be autoremoved depending on config.

P.S.: On a multi-arch system nearly every Depends is a choice even without or-groups: given that you e.g. pick banana:amd64 or banana:i386 for an M-A:foreign banana. And the t64 transition added a quadrillion of real vs. virtual bananas at least until everyone depends on bananat64.