When apt keeps back packages due to phased updates, it should list them separately

Bug #1988819 reported by Sergio Callegari
This bug affects 156 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Julian Andres Klode

Bug Description

After phased updates have been introduced, it may happen that apt upgrade shows packages as upgradable but ends up not upgrading them. In this case the packages are indicated as being "kept back".

Unfortunately, the feedback provided about this to the user is not very informative. The user sees the packages being kept back and thinks something is going wrong on the system.

When packages are kept back because of phased updates, apt should say so e.g., it should say that the upgrade is delayed.

Incidentally note that aptitude does not respect phased updates.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: apt 2.4.7
ProcVersionSignature: Ubuntu 5.15.0-47.51-generic 5.15.46
Uname: Linux 5.15.0-47-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: KDE
Date: Tue Sep 6 10:05:14 2022
EcryptfsInUse: Yes
InstallationDate: Installed on 2020-02-16 (933 days ago)
InstallationMedia: Kubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
SourcePackage: apt
UpgradeStatus: Upgraded to jammy on 2022-06-03 (94 days ago)

Revision history for this message
Sergio Callegari (callegar) wrote :
Revision history for this message
Seth Arnold (seth-arnold) wrote :

I have seen many people on IRC *very* upset after wasting a lot of time trying to install updates that apt will not let them install. Fixing this is critical to our reputation.


Revision history for this message
Julian Andres Klode (juliank) wrote :

We can't really store reasons for why something is kept back. And if we managed to do that, we'd end up with an untranslated line in the middle of the output for every non-English locale until it's translated and apt updated (it does not use language packs).

Revision history for this message
Brian Foster (blfoster) wrote :

Yeah, I was confused by this at first too. Here's a hint, try "apt-mark showhold":

   $ apt-mark showhold

If nothing is printed (as above), then the "issue" PRESUMABLY isn't local to your system; ergo, the cause is PROBABLY at the server, such as a phased update. Relax. It will probably sort itself out in a few days. (This hint is from https://askubuntu.com/questions/640986/how-to-get-a-list-of-installed-packages-held-back-from-upgrade "How to get a list of installed packages held back from upgrade?" (APT, etc., experts can please correct).)

As per above (#3), whilst the reason isn't stored, that does not mean apt(8)'s message cannot be improved. For example, instead of the current:

   The following packages have been kept back:

better might be something like:

   The following packages have been kept back:
   Some or all of those packages may be deliberately "on hold" on the server;
   please see manual page XXX(N) for information.

with (obviously) appropriate text in the manual page. (Exactly which man page is left to the experts.)

Nick Rosbrook (enr0n)
tags: added: foundations-triage-discuss
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
Revision history for this message
Seth Arnold (seth-arnold) wrote :

apt-cache policy knows when packages are phased; when apt needs to report that packages are held back, apt could look up each one to report phased status.

Not fixing this because the strings need translating is an argument for declaring APT a finished project and moving on to the Next Big Thing.

Revision history for this message
Robie Basak (racb) wrote :

> We can't really store reasons for why something is kept back.

How about:

When something is scored down because it is not phased, make a note. When the scoring is all done, take the list of held back packages, and identify all those that have a note against them. Then print (up to) two lists:

The following packages are being phased and have been kept back:

The following [additional] packages have been kept back:

(use two translatable strings for the second message, adding "additional" if the first list was non-zero).

Of course the exact strings can be adjusted, but I hope you see my point in fixing the "we don't know what final reason is because the different scoring adjustments get conflated" problem.

> And if we managed to do that, we'd end up with an untranslated line in the middle of the output for every non-English locale until it's translated and apt updated (it does not use language packs).

We should certainly be able to fix this in the development release and get translations in the usual way. For SRU, we could wait for translations, or not, but no need to block on a fix in the development release for this.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

An alternative, proposed by user avih on IRC, is to not report any of these packages to the user at all:

<avih> however, these phased updates are quite a big list which adds a lot of noise to my regular dist-upgrade, and it interferes with me reviewing what's about to get updated
<avih> the kept back list is quite bigger than the list of things to update...
<avih> arraybolt3: if this is indeed the standard order of things, why am i being shown at all what it's NOT going to install for reasons not related to errors or conflicts?

I can see a lot of appeal to not telling the user information -- from their perspective, the packages don't actually exist yet.

Maybe it'll cause confusion if of two machines sitting right next to each other, one can see the updates and the other cannot. That's not ideal.

But holding information back from the user doesn't require new strings, and casual users with one machine might never notice.

It's just fun to see an alternative idea that's 180 degrees different from my initial thought. :)

tags: added: foundations-todo
removed: foundations-triage-discuss
Revision history for this message
Julian Andres Klode (juliank) wrote :

I want to say I agree with avih and had the same idea but I had no chance to update the bug yet. Currently sorting out actual upgrade failures that might be regressions from the new phasing though.

It remains to be seen if we can do this safely. We need to add a new flag "PhasedKeep" or something to the depcache and then make sure to always clear this if we do not keep it.

Revision history for this message
mkoniecz (matkoniecz) wrote :

How can I simply disable phased packages? (I was hit numerous times by outdated packages, never had problem with update causing problems so for me this just makes things worse anyway)

Changed in apt (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Triaged
importance: Medium → High
assignee: nobody → Julian Andres Klode (juliank)
summary: - When apt keeps back packages due to phased updates, it should say so
+ When apt keeps back packages due to phased updates, it should say
+ nothing
Revision history for this message
Sergio Callegari (callegar) wrote : Re: When apt keeps back packages due to phased updates, it should say nothing

As the original reporter, I am a bit unconvinced by this change of perspective in how to fix the issue. I agree that being silent is easier, but does not feel quite the right thing to do for me, at least unless an option or something is added to see which upgrades are in progress and phased back.

- It is untrue that from the user perspective these packages do not exist yet, as suggested.

> from their perspective, the packages don't actually exist yet.

The packages do exist and can be quite visible to the user. If you `apt install`, it is the most recent version of the packages that gets installed.

- It is generally considered bad practice to open bugs without first trying the most recent version of some troublesome software

Hiding the phased out packages takes away the opportunity to users who are experiencing problems to try the most recent version before opening a bug.

- When neighboring machines start running different versions of the software and the user has the belief that they should be aligned because they use the same distribution, if trouble occurs the user may be lead to wrong conclusions (e.g. inculpate the hardware).

Again the user should be able to easily check if there is the opportunity to try the last version of something when that something is giving trouble.

- Finally, I am afraid that being non-transparent may lead users to develop non standard hacks to see (and maybe immediately install) the phased out packages. Incidentally, currently `aptitude upgrade` installs the phased out packages.

Revision history for this message
Julian Andres Klode (juliank) wrote :

If phasing upgrades are hidden in upgrade output then the output shows you relevant info to the update only which is nice. I kind of feel like `apt upgrade` and friends should show you only unexpected things. Like often people install 100 packages and then miss that they also remove 20 that they did not want to.

Phased updates are list listed as upgradable in `apt list --upgradable`, in the `?upgradable` pattern, and can be updated using the `install` command (e.g. `apt install a-phased-package`); the policy command tells you if it is indeed phasing or not, but not whether it's phasing for you.

I admit it's confusing, but the next iteration should be to hide those upgrades (as they were for months) and then we can iterate on a nicer interface for the other side cases.

Revision history for this message
Sergio Callegari (callegar) wrote :

Would it then be possible to:
1. silence `apt upgrade` wrt the kept back packages
2. update `apt list --upgradable` to say which upgradable packages are phased out (i.e. not upgraded right now by `apt upgrade`
3. Update the bug report guidelines suggesting to `apt install` the troublesome package and retrying before opening a bug
. update `aptitude` to behave as `apt` wrt to kept back packages?

Revision history for this message
Benjamin Schmid (benbuntu) wrote :

Even as a decade-old Ubuntu user this "The following packages have been kept back:" was a UX nightmare for me. It took me several runs of article reading, many unsuccessful "apt-mark showhold", breaking "apt install X" runs and still no clue at all about the background on these errors suddenly reappearing on 22.04 systems. I was this days old when I eventually was able to realize that this is all about "phased upgrades".

The implementation or effect from a UX is really utterly opaque. I'd go as far as to suggest: Refrain from phased upgrades until Ubuntu is able to signal apt users the reasons for packages held back!

In my decade-old history, "packages kept back" were 100% always some really, really bad borked stuff, i.e., on dist-upgrades. Always something to take immediate care about to not threat system integrity or automatic updates and hence threatening hence the system security. A reason why it never occurred to me to just ignore those line.

Revision history for this message
Gone fishing (humpmihk-msn) wrote (last edit ):

I've been using Ubuntu since Warty or Breezy, and I have had very few incidents of an update breaking the system. I think an update in Hardy broke the X server once. Getting to the point - this phased update system had me thinking something had gone horribly wrong with apt and the package database. Maybe the cure is not worse than the problem, but it appears to be.

I also "fixed" the problem by manually installing the held-back packages, which I can now see was a bad idea.

I would suggest that, as we now have apt messages:

# News about significant security updates, features and services will
# appear here to raise awareness and perhaps tease /r/Linux ;)
# Use 'pro config set apt_news=false' to hide this and future APT news.

Maybe added to the news could be something about phased updates such as:
# You may see packages being held back as part of phased updates - these packages are being held back to ensure system stability.
# If you want to turn off phased updates, create a file in /etc/apt/apt.conf.d, containing the following lines:


Revision history for this message
Seth Arnold (seth-arnold) wrote :

So far I've been arguing that apt should be more verbose about phasing, and why these packages are held back. A friend has suggested that instead apt should say *nothing*. I can see the appeal.


Revision history for this message
Adrien Nader (adrien-n) wrote :

The issue with being less verbose is that users will end up with the same issue when two neighbor machines have different updates. This also applies to machines belonging to different people as soon as these people discuss about a but that could be caused or solved by these updates.

I'd prefer to see a new line group which lists updates "rolling out over the next few days". It seems to be the same overall complexity to implement and it would be much much more informative. Hiding again seems like a step too far back to me.

Revision history for this message
Jeffrey Walton (noloader) wrote :

Yes, please fix.

From a security standpoint, we can't tell the "good" held backs (Phased Updates) from the "bad" held backs (problem with package). So we assume its a package problem, go in with a hammer, and use Aptitude's safe-upgrade to force the updates. Now we've subverted/undermined the Phased Updates when there was no problem.

Revision history for this message
Davison Long (number-one) wrote :

I for one don't like the idea of hiding the phased update info unless there is also a flag to show it. Not showing the info at all just obscures the issue even more and would lead to a lot of unnecessary confusion when odd situations might occur.

I also think it would be far better to have a separate non-translated message for packages held back due to phased updates and then add translations later. The current confusion is occurring world-wide; an English message about in process phased updates is going to provide a lot more useful info to a non-English speaker than what is happening now. Like others have mentioned, I also am a long-time Debian and Ubuntu user and this issue took WAY longer than it should have for me to understand what was actually going on since there was absolutely NO notices or info about phased updates at all in any apt commands or config files (except for apt policy on a specific package).

Revision history for this message
Walter (wdoekes) wrote :

Haha, yea, long time debian/ubuntu user here too. The only way I found out and know was because my script that parses apt-policy failed [1]. Pure luck [2].

Also seconding the do-translations-later. I don't know if there are statistics available for this, but based on anecdotal evidence alone, I say that many people run their OS in English anyway. And that the average debian/ubuntu user is used to more English than average users of proprietary operating systems..

[1] https://github.com/ossobv/vcutil/commit/9a34b973491f7165790f8c239f1b1c39bf4d1687
[2] Yay for defensive coding.

Revision history for this message
Mehmet Can Cömert (cancomert) wrote :

My current user experience is like following:
I just updated kept-back packages by mistake today by using apt install command.
I thought it was due to a wrong mark of Manuel tag in apt and marked them with apt-mark auto again.
Then after a bit search in internet I realized that it is due to a phased update policy, which makes sense.

I think either message should not shown at all (as suggested above)
or if you want to show the list, please state somehow that this is only info and we do not have to do something about it, such as:
The following packages have been kept back as a part of phased update policy of Ubuntu:

Just stating packages are kept back, gives the impression that something is broken in the dependency resolution.

Revision history for this message
lochfynne (lochfynne) wrote :

I totally agree with the previous writer.
Until reading this article, I manually upgraded Kept-back packages.

I support the idea of phased upgrades, but please create a separate notification stating something like:
The following packages have been kept back as a part of phased upgrade policy of Ubuntu: <list of packages affected>

This allows the user to deal with any other package issues separately.

Revision history for this message
Lin Manfu (linmanfu) wrote :

I was also confused by this change in behaviour, even though I knew about Phased Updates. I don't often update with apt now and had just forgotten about them.

I support the solution of listing phased updates separately.

I do not think the translation issue should hold this back. If apt has not internationalisation capability, then presumably other apt text such as "The following packages were automatically installed and are no longer required:" is not translated. Nor is the "News about significant security updates..." message is not translated either (and that contains a joke about Reddit, which is even more likely to confuse non-English speakers). And many non-English speakers will be very familiar with the process of copying a message they don't fully understand into a search engine to get more information and/or translation (maybe not an option for server users but we can expected them to use man apt).

summary: - When apt keeps back packages due to phased updates, it should say
- nothing
+ When apt keeps back packages due to phased updates, it should list them
+ separately
Revision history for this message
Sergio Callegari (callegar) wrote :

There is a further problem, I do not know if it should be a separate issue.

The updates icon in the system tray now appears also when there are in fact no updates that can be applied. Don't know if this actually depends on the presence of updates that are phased out or on the presence of Ubuntu Pro (esm-apps) that are not enabled.

Revision history for this message
Sergio Callegari (callegar) wrote :

After further investigation, I would say that "Discover" wants to upgrade a package (libmm-glib0) notwithstanding the fact that it is phased and that apt does not want to upgrade it.

Funny enough, Discover does not indicate the other phased packages (it is up to 24!) as upgradable.

Revision history for this message
Nathan Collins (ntc2) wrote :

My vote is to remove this "phased updates" feature until the UI can be fixed. Right now I run `apt update` and it tells me "N packages can be upgraded", then I run `apt upgrade` and it tells me no packages will be upgraded because "N packages are held back". There is no mention of "phased updates". Very confused, wasting my time on this after upgrading to 22.04. Have been using Ubuntu since 2005 and found this very confusing.

Revision history for this message
Xiao Wan (xwcal) wrote :
Download full text (3.8 KiB)

To add a data point and save others from preventable headaches, I encountered a related bug and reported it here:

If anyone cares, here are my two cents regarding phased updates:

Or if lack of markdown formatting is not an issue, here is goes:

# A Short Critique of Phased Updates

Some thoughts after I stumbled on phased updates during a recent migration from Bionic to Jammy:

## Randomness and Repeatability

Per [Debian repository specs](https://wiki.debian.org/DebianRepository/Format#Phased-Update-Percentage), randomization is achieved by the following means:

To determine whether an update is applicable to a given machine, a client shall seed a random number generator (APT uses std::minstd_rand) with the string:


where sourcePackage is the name of the source package, version the version, and machineID the contents of /etc/machine-id.

It shall then extract an integer from a uniform integer distribution between 0 and 100.

To the uninitiated: this is not how a random number generator is meant to be used. If you want a repeatable sequence of random numbers, you seed the RNG once and then draw the desired number of samples. You don't reseed the RNG for each new sample, since if your seeds are already random, then the RNG step is pointless, whereas if your seeds aren't random, say, in an extreme case, if you always seed with the same number, your "random" number is guaranteed to be the same each time.

To achieve repeatable randomness with respect to both the package and machine, there is an obviously superior choice in terms of both randomness and portability: **just use a cryptographic hash like sha256**.

This begs the question of how repeatable the design really is.
To quote [https://wiki.debian.org/MachineId](https://wiki.debian.org/MachineId):

what is the machine id actually used for?

    a comprehensive list is probably not possible without grepping the code

That's to say, although for repeatable experiments it's foreseeably necessary to mess with the machine ID or share it publicly,
the consequences of such tinkering or disclosure are hard to assess, since nobody knows for sure how and where the machine ID is and will be used.

This leads to an easily conceivable alternative: since apt only uses the machine ID for the purpose of repeatable randomization, how about designating another user configurable value stored in, say `/etc/apt/phased-update-random-seed`, only for this particular purpose and by default initialized to the machine ID?

## User Choice

From [this response](https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2017399/comments/1) to my bug report (one question I forgot to ask is why phase security updates at all) and a comment on [askubuntu](https://askubuntu.com/questions/1431940/what-are-phased-updates-and-why-does-ubuntu-use-them) that doesn't cite any source, it does appear that consideration has been given regarding the need to treat different packages (at least different classes of packages) differently when it comes to phasi...


Revision history for this message
Julian Andres Klode (juliank) wrote :

I did not make any changes to the design when moving it from Update-Manager in apt, it's been this way for 10 years now, it's a proven concept.

The initial implementation used and md5 but the math gets complex when dealing with 128 bit integers.

You can easily override the machine-id for apt by setting the apt.conf option, /etc/machine-id is a default. It's general consensus that everything should be using the same machine id though.

Revision history for this message
Xiao Wan (xwcal) wrote :

@juliank Thanks for the tip about machine id. May I also ask, for repeatability, is there an official/supported way to run (unit) tests on the upgrade logic alone, say, by feeding `/var/lib/dpkg/status`, index files and maybe some environment variables/APT::* preferences to `apt-get` and then running a simulation?

As you have seen in https://bugs.launchpad.net/ubuntu/+source/apt/+bug/2017399, the least amount of state for a reproducible report involved a livecd iso. And if I had been much further down the road it would have been impossible to trace the issue to the "original" state that everybody else also has access to.

Regarding md5, the first or last 64 bits (or 32bits for i386) already give a good random number for this purpose. There is no need to deal with 128bit integers.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Generally for reproducing issues I set Phased-Update-Percentage to 0 on the packages the user had phased so it's independent of machine-id, it's not been a problem in my experience to reproduce issues.

Revision history for this message
Julian Andres Klode (juliank) wrote :

One thing I just noticed is that if you have something like gnome-shell and mutter where gnome-shell depends on the new mutter and only mutter is "not for us" we end up with something like:

The following packages have been kept back:
  gnome-shell gnome-shell-common mutter

with phasing specific message this would look like this:

The following packages have been kept back as they are phasing:
The following packages have been kept back:
  gnome-shell gnome-shell-common

Because we kept back mutter and it is phasing, but the gnome-shell update is not anymore, it is kept back because it has broken dependencies on the mutter that is phasing.

Revision history for this message
Roger Tawa (rogertawa) wrote :

Julian, if apt knows that mutter is being held back due to phasing, and that gnome-shell and gnome-shell-common are being held back because of the mutter dependency, then it should be possible for the message to look something like:

  The following packages have been kept back as they are phasing or depend
  on a phased package:
    gnome-shell gnome-shell-common mutter

Revision history for this message
Julian Andres Klode (juliank) wrote :

It doesn't know that, no. It can guess that a phased package was kept back due to phasing, but the solvers don't track reasons for any decisions so it can't go past that.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.