apt errors during apt-get install distro-info prevent controller upgrade from completing

[Trent Lloyd]

While working on a controller upgrade from 3.3.0->3.3.1, the controller was stuck during startup repeating the following message every 10s:
2024-02-20 06:09:55 INFO juju.packaging.manager run.go:102 Retrying: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install distro-info

machine agents were unable to connect as the upgrade was still in progress. The reason that this was repeating is because apt had a dependency error on the controller, even though distro-info was actually installed already:
root@saving-badger:~# apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install distro-info; echo $?
Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 linux-headers-5.15.0-91-generic : Depends: linux-headers-5.15.0-91 but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
100

This apt error also happened automatically (without manual intervention) during unattended-upgrades a couple of months previous, as the controller had previously run out of disk space (due to logs filling the disk) and during this unattended-upgrades run linux-headers-5.15.0-91-generic had installed but linux-headers-5.15.0-91 failed to unpack due to the out of space situation and apt got stuck that way, leaving it in a broken state. So it's not a case of a user manually creating a problem.

The controller was happily running even after a reboot in this broken apt state, the issue only appeared after the upgrade to 3.3.1.

The process to install this package could possibly be more robust to such a failure and not prevent an upgrade - especially if the package is actually installed. Perhaps by checking if the package is already installed some other way, rather than relying on the apt-get error code.

Alternatively if that's not acceptable some kind of better error may need to be logged to make it more clear to the user that this needs manual fixing.

This is similar but not the same as Bug #2011637 where the same command hangs due to dpkg prompting a question, due to a lack of noninteractive.