[focal] cannot resolve dependency when installing a package depending on 'chrony' since 245.4-4ubuntu3.21

[Thomas Faivre]

Hello,

Since the publishing of version 245.4-4ubuntu3.21 on focal-updates, we cannot install our product.
Our product depends on specifically chrony for the time-daemon, but apt cannot resolve the dependency anymore when systemd 245.4-4ubuntu3.20 is installed.

It might be specific to systemd-timesyncd, or an issue with apt itself but I am not sure.

Reproduced using the "hello" package by modifying its dependencies:

> root@ubuntu2004:~# apt download hello
> Get:1 http://mirrors.ubuntu.com/mirrors.txt Mirrorlist [71 B]
> Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 hello amd64 2.10-2ubuntu2 [28.2 kB]
> Fetched 28.3 kB in 0s (95.6 kB/s)
> W: Download is performed unsandboxed as root as file '/root/hello_2.10-2ubuntu2_amd64.deb' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
> root@ubuntu2004:~# dpkg-deb -R hello_2.10-2ubuntu2_amd64.deb hello
> root@ubuntu2004:~# sed -i -e '/Depends/s/$/, chrony/' hello/DEBIAN/control
> root@ubuntu2004:~# dpkg -b hello/
> dpkg-deb: building package 'hello' in 'hello.deb'.
> root@ubuntu2004:~# apt -y install ./hello.deb
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Note, selecting 'hello' instead of './hello.deb'
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
> hello : Depends: chrony
> E: Unable to correct problems, you have held broken packages.

The issue can be fixed by upgrading systemd to 245.4-4ubuntu3.21:
> root@ubuntu2004:~# apt -y install systemd
[...]
> Setting up systemd (245.4-4ubuntu3.21) ...
> Setting up systemd-timesyncd (245.4-4ubuntu3.21) ...
[...]
> root@ubuntu2004:~# apt -y install ./hello.deb
[...]
> The following packages will be REMOVED:
> systemd-timesyncd
> The following NEW packages will be installed:
> chrony hello
[...]

But that solution is a bit annoying as our customers might not want to upgrade systemd for some reason.

OR by manually installing chrony:
> root@ubuntu2004:~# apt -y install chrony
[...]
> The following packages will be REMOVED:
> systemd-timesyncd
> The following NEW packages will be installed:
> chrony
[...]
> root@ubuntu2004:~# apt -y install ./hello.deb
[...]
> The following NEW packages will be installed:
> hello
[...]

Conflict with enabled debug logs:
> root@ubuntu2004:~# apt -o Debug::pkgProblemResolver=1 install ./hello.deb
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Note, selecting 'hello' instead of './hello.deb'
> Starting pkgProblemResolver with broken count: 1
> Starting 2 pkgProblemResolver with broken count: 1
> Investigating (0) hello:amd64 < none -> 2.10-2ubuntu2 @un puN Ib >
> Broken hello:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
> Considering chrony:amd64 1 as a solution to hello:amd64 9999
> Re-Instated chrony:amd64
> Investigating (0) systemd-timesyncd:amd64 < 245.4-4ubuntu3.20 -> 245.4-4ubuntu3.21 @ii umU Ib >
> Broken systemd-timesyncd:amd64 Conflicts on time-daemon:amd64 < none @un H >
> Conflicts//Breaks against version 1:4.2.8p12+dfsg-3ubuntu4.20.04.1 for ntp but that is not InstVer, ignoring
> Considering chrony:amd64 1 as a solution to systemd-timesyncd:amd64 14
> Added chrony:amd64 to the remove list
> Conflicts//Breaks against version 1:6.2p3-4 for openntpd but that is not InstVer, ignoring
> Conflicts//Breaks against version 1.1.8+dfsg1-4build1 for ntpsec but that is not InstVer, ignoring
> Conflicts//Breaks against version 1:4.2.8p12+dfsg-3ubuntu4 for ntp but that is not InstVer, ignoring
> Conflicts//Breaks against version 3.5-6ubuntu6 for chrony but that is not InstVer, ignoring
> Fixing systemd-timesyncd:amd64 via keep of chrony:amd64
> Investigating (1) hello:amd64 < none -> 2.10-2ubuntu2 @un puN Ib >
> Broken hello:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
> Considering chrony:amd64 1 as a solution to hello:amd64 9999
> Considering chrony:amd64 1 as a solution to hello:amd64 9999
> Considering maas:amd64 0 as a solution to hello:amd64 9999
> Re-Instated apparmor:amd64
> Re-Instated liblzo2-2:amd64
> Re-Instated squashfs-tools:amd64
> Re-Instated dbus-user-session:amd64
> Re-Instated snapd:amd64
> Re-Instated maas:amd64
> Investigating (1) maas:amd64 < none -> 1:0.7 @un uN Ib >
> Broken maas:amd64 Conflicts on isc-dhcp-server:amd64 < 4.4.1-2.1ubuntu5.20.04.5 @ii mK >
> Considering isc-dhcp-server:amd64 0 as a solution to maas:amd64 0
> Holding Back maas:amd64 rather than change isc-dhcp-server:amd64
> Investigating (2) hello:amd64 < none -> 2.10-2ubuntu2 @un puN Ib >
> Broken hello:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
> Considering chrony:amd64 1 as a solution to hello:amd64 9999
> Considering chrony:amd64 1 as a solution to hello:amd64 9999
> Considering maas:amd64 0 as a solution to hello:amd64 9999
> Considering maas:amd64 0 as a solution to hello:amd64 9999
> Done
> Some packages could not be installed. This may mean that you have
> requested an impossible situation or if you are using the unstable
> distribution that some required packages have not yet been created
> or been moved out of Incoming.
> The following information may help to resolve the situation:
>
> The following packages have unmet dependencies:
> hello : Depends: chrony
> E: Unable to correct problems, you have held broken packages.

Thank you for any help you can provide on that matter!

[Thomas Faivre (thomas-faivre)]

Hello again,

New info on this issue.
I was trying to reproduce on ARM (and I did), but there is a but. And it's kind of an important one.

Our ARM VMs had version 245.4-4ubuntu3.19 installed, and the issue was not seen. When upgrading to 245.4-4ubuntu3.20, the issue appears.

So it seems that the issue is upgrading *from* 245.4-4ubuntu3.20, and not *to* 245.4-4ubuntu3.21.

I don't know what can be done then, maybe remove the version 245.4-4ubuntu3.20 from repositories? It might be a bit aggressive...

But I am still unsure why an available version upgrade of a package creates a conflict...

Any suggestions?

Thanks!

[Nick Rosbrook (enr0n)]

I have not had time to look at this more closely, but it does seem unlikely that this is caused by either systemd update, because they do not change anything in the systemd packaging itself (each update only added new patches against upstream source).

For now I will mark apt as affected too in case someone else has time to look at this before I can circle back.

[Julian Andres Klode (juliank)]

This is the expected behavior of apt, and a deeply integral part of the solver algorithm, that it does not try to remove packages that have an upgrade available. So that's not something I'd want to touch because the regression potential is huge.

You can always use pass systemd-timesyncd- to apt install to help it along that you want to remove that package.

[Thomas Faivre (thomas-faivre)]

Hello,

thanks for the update!
I understand the reluctance to change something like that. But I still don't understand why the issue is not present when the version 245.4-4ubuntu3.19 is installed instead of 245.4-4ubuntu3.20.

> root@ubuntu2004:~# apt list --installed systemd
> Listing... Done
> systemd/now 245.4-4ubuntu3.19 amd64 [installed,upgradable to: 245.4-4ubuntu3.21]
> root@ubuntu2004:~# apt install ./hello.deb
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Note, selecting 'hello' instead of './hello.deb'
> The following additional packages will be installed:
> chrony libnss-systemd libpam-systemd libsystemd0 systemd systemd-sysv
> Suggested packages:
> systemd-container policykit-1
> The following packages will be REMOVED:
> systemd-timesyncd
> The following NEW packages will be installed:
> chrony hello
> The following packages will be upgraded:
> libnss-systemd libpam-systemd libsystemd0 systemd systemd-sysv
> 5 upgraded, 2 newly installed, 1 to remove and 73 not upgraded.
> Need to get 4,591 kB/4,619 kB of archives.
> After this operation, 416 kB of additional disk space will be used.
> Do you want to continue? [Y/n] ^C

I would expect apt to not remove systemd-timesyncd according to what you are saying, since versions 245.4-4ubuntu3.20 and 245.4-4ubuntu3.21 are available.
Also, why can I install chrony manually in that case?

I feel like there is still something fishy somewhere...

[Julian Andres Klode (juliank)]

There's more complexity of course because I only talk about the first of the two solvers apt runs. The first solver marks the changes based on following the graph from the user-initiated actions downward to dependencies, marking Depends for install, Conflicts for remove, and so on.

If that produces any broken packages, the second solver tries to heuristically solve them. It's code has grown over 20 years and we can't reason much about it.

Installing chrony manually is somewhat stronger than automatically installing it in the scoring for the solver, shifting the fates a bit.

I plan to eventually return to work on a state of the art solver, which will solve all issues (you know what I mean), but so far am lacking time. I've written a bunch of solvers, or problem converters, using z3 and clasp as backends, but I'm struggling a bit with solver runtime, explaining why solving failed, and handling order expectations - more research is needed. Usually those happen in my free time during the christmas break.