phased updates API and client for aptitude

[Lenin]

Having the configuration at
https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345/24
it fails to work.

The phased updates get distributed on some hosts anyways for 22.04.

Please properly document and test this, and provide a working configuration to opt out from phased updates.

[Julian Andres Klode (juliank)]

I do not believe there is a bug in apt here. APT literally is too strict now, it fails to install packages. It will get more relaxed in the coming weeks.

Anyway, if you find out where those get upgraded, please reopen the bug and reassign to the correct package.

[Lenin (gagarin)]

Do you have a suggestion where to assign the bug if not to src:apt bin:apt-doc if not these?

[Lenin (gagarin)]

I get these updates, which I'd prefer not to be part of phased updates:

   libapt-pkg6.0:amd64 (2.5.1)
   libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.1)
   libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.1)
   libqt5opengl5-dev:amd64 (5.15.3+dfsg-2ubuntu0.1)
   libqt5opengl5:amd64 (5.15.3+dfsg-2ubuntu0.1)
   libqt5sql5:amd64 (5.15.3+dfsg-2ubuntu0.1)
   libsmbclient:amd64 (2:4.15.5~dfsg-0ubuntu5.1)
   libsss-certmap0 (2.6.3-1ubuntu3.1)
   libsss-idmap0 (2.6.3-1ubuntu3.1)
   libsss-nss-idmap0 (2.6.3-1ubuntu3.1)
   qemu-user-static (1:6.2+dfsg-2ubuntu6.3)
   qemu-utils (1:6.2+dfsg-2ubuntu6.3)
   qt5-qmake:amd64 (5.15.3+dfsg-2ubuntu0.1)
   qtbase5-dev:amd64 (5.15.3+dfsg-2ubuntu0.1)
   samba-common-bin (2:4.15.5~dfsg-0ubuntu5.1)
   samba-libs:amd64 (2:4.15.5~dfsg-0ubuntu5.1)
   sssd-ad (2.6.3-1ubuntu3.1)
   sssd-ad-common (2.6.3-1ubuntu3.1)
   sssd-common (2.6.3-1ubuntu3.1)
   sssd-dbus (2.6.3-1ubuntu3.1)
   sssd-ipa (2.6.3-1ubuntu3.1)
   sssd-krb5 (2.6.3-1ubuntu3.1)
   sssd-krb5-common (2.6.3-1ubuntu3.1)
   sssd-ldap (2.6.3-1ubuntu3.1)
   sssd-proxy (2.6.3-1ubuntu3.1)
   sssd-tools (2.6.3-1ubuntu3.1)

fits to:

https://people.canonical.com/~ubuntu-archive/phased-updates.html
(which sometimes lags behind 6 hours, depending on mirror and sync times)

[Julian Andres Klode (juliank)]

The apt update is from kinetic, the development release. What are you doing?

The question is what installs them. Certainly it's not apt itself doing that unless you have very weird pinning in place.

[Julian Andres Klode (juliank)]

For concrete details:

1) Which command would install these updates
2) Please attached `apt policy ...names of affected packages...` output
3) Attach your sources.list and sources.list.d files
4) Attach your preferences and preferences.d files

Maybe just throw all the files into a tarball.

[Lenin (gagarin)]

1) We have like 100+ Ubuntu 22.04 machines, the updates get installed with aptitude (via aptitude-robot). No pinning, but one held package (nvtop) (on 66% of machines).

2) Here's just one package example:
$ apt policy samba-libs
samba-libs:
  Installed: 2:4.15.5~dfsg-0ubuntu5.1
  Candidate: 2:4.15.5~dfsg-0ubuntu5.1
  Version table:
 *** 2:4.15.5~dfsg-0ubuntu5.1 500 (phased 40%)
        500 http://ubuntu.ethz.ch/ubuntu jammy-updates/main amd64 Packages
        500 http://ubuntu.ethz.ch/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:4.15.5~dfsg-0ubuntu5 500
        500 http://ubuntu.ethz.ch/ubuntu jammy/main amd64 Packages

3) $ cat sources.list
deb http://ubuntu.ethz.ch/ubuntu jammy main restricted universe multiverse
deb-src http://ubuntu.ethz.ch/ubuntu jammy main restricted universe multiverse
deb http://ubuntu.ethz.ch/ubuntu jammy-security main restricted universe multiverse
deb-src http://ubuntu.ethz.ch/ubuntu jammy-security main restricted universe multiverse
deb http://ubuntu.ethz.ch/ubuntu jammy-updates main restricted universe multiverse
deb-src http://ubuntu.ethz.ch/ubuntu jammy-updates main restricted universe multiverse
deb http://ubuntu.ethz.ch/ubuntu jammy-proposed main restricted universe multiverse
deb-src http://ubuntu.ethz.ch/ubuntu jammy-proposed main restricted universe multiverse

sources.list.d 404 (all empty)

4) nothing there...

It's so little, I just copy pasted it here. Hope that's fine.

[Lenin (gagarin)]

also we noticed that while the status page lags: https://people.canonical.com/~ubuntu-archive/phased-updates.html we already get phased updates that are not listed:
Packages not installed from apt repositories (1):
   libmysqlclient21:amd64 (8.0.29-0ubuntu0.22.04.3)

reported by xymon monitoring system, hobbit plugins, apt plugin specifically.

apt policy confirms it is part of phased updates.

this problem exists with apt 2.4.6 as well as 2.5.1 on Ubuntu 22.04

[Lenin (gagarin)]

a downgrade is possible:

hostname:~# apt policy libmysqlclient21
libmysqlclient21:
  Installed: 8.0.29-0ubuntu0.22.04.3
  Candidate: 8.0.29-0ubuntu0.22.04.3
  Version table:
 *** 8.0.29-0ubuntu0.22.04.3 500 (phased 10%)
        500 http://ubuntu.ethz.ch/ubuntu jammy-updates/main amd64 Packages
        500 http://ubuntu.ethz.ch/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     8.0.29-0ubuntu0.22.04.2 500
        500 http://ubuntu.ethz.ch/ubuntu jammy-security/main amd64 Packages
     8.0.28-0ubuntu4 500
        500 http://ubuntu.ethz.ch/ubuntu jammy/main amd64 Packages
hostname:~# apt install libmysqlclient21=8.0.29-0ubuntu0.22.04.2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libaom-dev libarmadillo-dev libarpack2-dev libavformat-dev libblosc-dev libcharls-dev libdav1d-dev
  libde265-dev libdouble-conversion-dev libfreexl-dev libfyba-dev libgeos-dev libgeotiff-dev libgl2ps-dev
  libglew-dev libheif-dev libjson-c-dev libjsoncpp-dev libkml-dev libkmlconvenience1 libkmlregionator1
  libkmlxsd1 liblapack-dev libnetcdf-c++4 libnetcdf-cxx-legacy-dev libogdi-dev libopenjp2-7-dev libopenni2-0
  libopenni2-dev libpcl-apps1.12 libpcl-common1.12 libpcl-features1.12 libpcl-filters1.12 libpcl-io1.12
  libpcl-kdtree1.12 libpcl-keypoints1.12 libpcl-ml1.12 libpcl-octree1.12 libpcl-outofcore1.12 libpcl-people1.12
  libpcl-recognition1.12 libpcl-registration1.12 libpcl-sample-consensus1.12 libpcl-search1.12
  libpcl-segmentation1.12 libpcl-stereo1.12 libpcl-surface1.12 libpcl-tracking1.12 libpcl-visualization1.12
  libpoppler-dev libpoppler-private-dev libproj-dev librttopo-dev libspatialite-dev libsuperlu-dev
  libswscale-dev libtheora-dev liburiparser-dev libusb-1.0-0-dev libutfcpp-dev libvtk9-java libvtk9.1-qt
  libx265-dev libzstd-dev python3-vtk9 qttools5-private-dev tcl-dev tcl8.6-dev tk-dev tk8.6-dev vtk9
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  default-libmysqlclient-dev libgdal-dev libmysqlclient-dev libpcl-dev libvtk9-dev libvtk9-qt-dev
The following packages will be DOWNGRADED:
  libmysqlclient21
0 upgraded, 0 newly installed, 1 downgraded, 6 to remove and 0 not upgraded.
Need to get 1,273 kB of archives.
After this operation, 102 MB disk space will be freed.
Do you want to continue? [Y/n]

hostname:~# apt install libmysqlclient21
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libaom-dev libarmadillo-dev libarpack2-dev libavformat-dev libblosc-dev libcharls-dev libdav1d-dev
  libde265-dev libdouble-conversion-dev libfreexl-dev libfyba-dev libgeos-dev libgeotiff-dev libgl2ps-dev
  libglew-dev libheif-dev libjson-c-dev libjsoncpp-dev libkml-dev libkmlconvenience1 libkmlregionator1
  libkmlxsd1 liblapack-dev libnetcdf-c++4 libnetcdf-cxx-legacy-dev libogdi-dev libopenjp2-7-dev libopenni2-0
  libopenni2-dev libpcl-apps1.12 libpcl-common1.12 libpcl-features1.12 libpcl-filters1.12 libpcl-io1.12
  libpcl-kdtree1.12 libpcl-keypo...

[Lenin (gagarin)]

I wasn't expecting different behaviour of apt, apt-get, aptitude, and here's proof they're all linked against same libapt, same version:

hostname:~# ldd $(which apt-get) |grep apt
 libapt-private.so.0.0 => /lib/x86_64-linux-gnu/libapt-private.so.0.0 (0x00007fe7c1c68000)
 libapt-pkg.so.6.0 => /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 (0x00007fe7c1aa6000)
hostname:~# ldd $(which apt) |grep apt
 libapt-private.so.0.0 => /lib/x86_64-linux-gnu/libapt-private.so.0.0 (0x00007fd3023a5000)
 libapt-pkg.so.6.0 => /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 (0x00007fd3021e3000)
hostname:~# ldd $(which aptitude) |grep apt
 libapt-pkg.so.6.0 => /lib/x86_64-linux-gnu/libapt-pkg.so.6.0 (0x00007fa63a117000)

[Julian Andres Klode (juliank)]

In your libmysqlclient21 example everything seems to be working correctly so I don't understand why you post that.

- you are eligible for 8.0.29-0ubuntu0.22.04.3
- you force downgrade to 8.0.29-0ubuntu0.22.04.2
- you upgrade once again to 8.0.29-0ubuntu0.22.04.3
- no higher version available in final dist-upgrade run

[Lenin (gagarin)]

I partly agree, the problem seems it is a phased update? Can you confirm or not?
Because other 22.04 machines don't get that version (90% of the other 22.04 machines).

Thus I wish to have a way to not take part of phased updates, especially since we disable
auto bug reporting (which seems to be crucial part of phased updates roll out increment or roll back/discard)

[Lenin (gagarin)]

it appears the phased updates only get installed if sources.list also has proposed repos, without them, it's all fine. sorry for the noise

[Lenin (gagarin)]

it appears phased upates are everywhere and not just with proposed repos... so although the bug might be against the wrong package filed (feel free to reassign to somewhere else). there's a bug
lacking information how to properly opt out of phased updates.

which is pointless to take part of if the automatic bug reporting is disabled/removed.

[Julian Andres Klode (juliank)]

It's documented in the discourse thread (Discourse is where Ubuntu documentation lives) You have not provided evidence that it does not work for you. APT itself has test cases that proof that *it does* work.

Note that newer apt versions (2.4.6, 2.5.1) will install more phased updates

- as phasing only applies to upgrades, not to new installs, so if you `apt install phased`, you'll get the phased version, regardless of your setting,
- or if there is a phased=1 security update and a phased=2 update, you'll get phased=2 even if it's phasing at 0.

You can restore previous behavior by setting APT::Get::Phase-Policy to true; but note that this can cause the `apt install` command to fail to install packages, see bug 1979244 for more details.

All other phased updates should show as being kept back in the upgrade output.

If you want to move this further you'll have to provide one of the following:

- A reproducer
- A log that shows a phased update being installed that should not be, including the command that installed the update and any debug output

Either must be accompanied by a tarball containing:

- /etc/apt
- /var/lib/dpkg/status
- /var/lib/apt

You also want to check yourself that

- `apt-config dump` shows `APT::Get::Never-Include-Phased-Updates "false";`
- `apt-config dump` shows no entry, or a negative one, for `APT::Get::Always-Include-Phased-Updates`

If you use update-manager it's not entirely clear how this interacts with the new phasing code in 2.4.6, so you might also/instead have to set

`Update-Manager::Never-Include-Phased-Updates "false";`

As documented in the discourse post.

[Julian Andres Klode (juliank)]

I recall/see again a mention of aptitude:

aptitude and aptitude-robot are not supported components. They are packages provided by the community as part of the universe repository.

While the implementation of phased updates in apt versions prior to 2.4.6 would work there, the current version does not. Implementing phased updates in aptitude for the current algorithm is out of scope, as aptitude does not use the apt resolver where the keep-back is implemented, and a generic API to handle phasing is not yet exposed in apt.

Switching to the policy based pinning as documented before should make aptitude respect pinning more, however, aptitude is not a reliable tool that respects your choices like that (its solver will happily suggest solutions violating your wishes; it also generally is too happy to remove packages), so you do not want to use it in an automated setting.

[Julian Andres Klode (juliank)]

I have added an aptitude task and set the tasks accordingly to indicate that this is a feature request for aptitude. The needed API will surface at some point in APT due to some refactoring, likely in the 23.04 cycle.

aptitude could then consume that API in its resolver and make choices to keep back phased upgrades, however, that is up to the aptitude maintainers upstream. Do note that aptitude is mostly unmaintained these days.

Alternatively, aptitude can also enable the policy based variant as that likely works for its solver. That could be SRUed into jammy if somebody wants to provide a patch for that, do the SRU paperwork, and get it sponsored and then work on getting it released. Do note that this option is experimental and may disappear in a future release as we finalize the correct approach.

[Axel Beckert (xtaran)]

Hi Julian,

Julian Andres Klode wrote:
> aptitude is not a reliable tool that respects your choices like that
> (its solver will happily suggest solutions violating your wishes; it

I have to object here. It very well respects the user's wishes and
also has a setting where the user can configure it should propose
solutions violating holds and forbids.

> also generally is too happy to remove packages),

Again, this solely depends on your settings.

  Regards, Axel
--
 ,''`. | Axel Beckert <email address hidden>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
  `- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE

[Julian Andres Klode (juliank)]

To make it clear, my suggestion for a workaround for jammy would be to have

_config->CndSet("APT::Get::Phase-Policy", true);

somewhere in the aptitude code. It is unclear to me if aptitude will break then as phased versions get their pin limited to 1, apt certainly does if there's a mismatch as in bug 1979244

Anyone doing the SRU will have to analyse that regression potential.

[Axel Beckert (xtaran)]

Hi Julian,

Julian Andres Klode wrote:
> Do note that aptitude is mostly unmaintained these days.

I have to object here as well:

There's no development currently, but it is _not_ unmaintained. (And
the current RC bug in Debian will soon be addressed. I'm just back
from holidays.)

  Regards, Axel
--
 ,''`. | Axel Beckert <email address hidden>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
  `- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE

[Lenin (gagarin)]

Hi Julian,

Looks like APT::Get::Phase-Policy true; did what I wanted. Thank you.

[Lenin (gagarin)]

hobbit-plugins

client/apt check reports:

 Fri Sep 23 08:59:06 2022 - apt NOT ok

Ubuntu 22.04.1 LTS

yellow Packages not installed from apt repositories (4):
   libnss-systemd:amd64 (249.11-0ubuntu3.4)
   libudev1:amd64 (249.11-0ubuntu3.4)
   systemd-sysv (249.11-0ubuntu3.4)
   systemd-timesyncd (249.11-0ubuntu3.4)

green Last apt update: 0.1 day(s) ago

without checking if packages were installed while a phased update is going on:
https://people.canonical.com/~ubuntu-archive/phased-updates.html

one workaround would be to patch the check with:
pkgreport('Packages not installed from apt repositories (maybe phased?)', 'green', @no_repo);

but that would blind us on seeing packages really not installed from packages (and not phased updates)

[Lenin (gagarin)]

so if anyone else wants to opt out from it, you're welcome:

https://github.com/alexmyczko/autoexec.bat/blob/master/config.sys/ubuntu-remove-phased-updates