install hook error: Unable to correct problems, you have held broken packages

[Drew Freiberger]

I ran into an issue with the nova-cloud-controller deployment that resulted in the following error repeating during install hook on a new lxd container on top of a new metal.

2021-07-20 20:25:25 WARNING unit.nova-cloud-controller/2.install logger.go:60 E: Unable to correct problems, you have held broken packages.
2021-07-20 20:25:26 ERROR juju.worker.uniter.operation runhook.go:139 hook "install" (via explicit, bespoke hook script) failed: exit status 100

Focal-Ussuri cloud, latest 21.04 n-c-c charm (#355).

The issue was simple to correct by running "juju ssh nova-cloud-controller/2 sudo apt-get update".

Interestingly, the next log line after the resolved hook status was:

2021-07-20 21:31:28 INFO unit.nova-cloud-controller/2.juju-log server.go:314 Installing python3-psutil with options: ['--option=Dpkg::Options::=--force-confold']

However, there's an ch_fetch.apt_update() called before installing that package.

But, I see that there are a few packages installed in the "install" hook shell file without first calling 'apt-get update' which may have been the blocker (along with something else in this new cloud deployment happening during install to block apt from having an up-to-date package repository list.)

My apt history.log shows python3-pip with a lot of pre-reqs installating via the install script just after I updated the apt repos, so I think the issue is the pre-reqs for python3-pip may have been missing from the repo cache due to prior build process failure (maybe proxy timeout or something similar).

Can we get an error-gated "apt-get update" included in the install hook script if "apt-get install" needs to be run to prevent this issue in the future?

[Drew Freiberger (afreiberger)]

/var/log/apt/history.log surrounding event:

Start-Date: 2021-07-20 20:24:18
Commandline: /usr/bin/apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet dist-upgrade
Upgrade: libpython3.8-minimal:amd64 (3.8.5-1~20.04.3, 3.8.10-0ubuntu1~20.04), libsystemd0:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), apt:amd64 (2.0.5, 2.0.6), linux-base:amd64 (4.5ubuntu3.1, 4.5ubuntu3.6), libpython3.8:amd64 (3.8.5-1~20.04.3, 3.8.10-0ubuntu1~20.04), python3.8:amd64 (3.8.5-1~20.04.3, 3.8.10-0ubuntu1~20.04), open-vm-tools:amd64 (2:11.0.5-4, 2:11.2.5-2ubuntu1~ubuntu20.04.1), udev:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), libapt-pkg6.0:amd64 (2.0.5, 2.0.6), libudev1:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), systemd-timesyncd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), libuv1:amd64 (1.34.2-1ubuntu1.1, 1.34.2-1ubuntu1.3), python3-distupgrade:amd64 (1:20.04.33, 1:20.04.35), ubuntu-release-upgrader-core:amd64 (1:20.04.33, 1:20.04.35), python3.8-minimal:amd64 (3.8.5-1~20.04.3, 3.8.10-0ubuntu1~20.04), systemd-sysv:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), libpam-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), apt-utils:amd64 (2.0.5, 2.0.6), libnss-systemd:amd64 (245.4-4ubuntu3.7, 245.4-4ubuntu3.10), libpython3.8-stdlib:amd64 (3.8.5-1~20.04.3, 3.8.10-0ubuntu1~20.04)
End-Date: 2021-07-20 20:24:33

Start-Date: 2021-07-20 20:24:34
Commandline: /usr/bin/apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install curl cpu-checker bridge-utils tmux ubuntu-fan
Install: msr-tools:amd64 (1.3-3, automatic), bridge-utils:amd64 (1.6-2ubuntu1), dnsmasq-base:amd64 (2.80-1.1ubuntu1.4, automatic), ubuntu-fan:amd64 (0.12.13), libidn11:amd64 (1.33-2.2ubuntu2, automatic), dns-root-data:amd64 (2019052802, automatic), cpu-checker:amd64 (0.7-1.1)
End-Date: 2021-07-20 20:24:37

Start-Date: 2021-07-20 20:24:48
Commandline: apt-get -y install python3-netaddr
Install: python3-netaddr:amd64 (0.7.19-3), ieee-data:amd64 (20180805.1, automatic)
End-Date: 2021-07-20 20:24:48

*HOOK FAILED at 20:24:49, meaning the next install after python3-netaddr triggered the condition*

Start-Date: 2021-07-20 21:31:05
Commandline: apt-get -y install python3-pip
Install: libmpc3:amd64 (1.1.0-1, automatic), python3-dev:amd64 (3.8.2-0ubuntu2, automatic), linux-libc-dev:amd64 (5.4.0-80.90, automatic), libpython3.8-dev:amd64 (3.8.10-0ubuntu1~20.04, automatic), gcc-9-base:amd64 (9.3.0-17ubuntu1~20.04, automatic), libfakeroot:amd64 (1.24-1, automatic), libc6-dev:amd64 (2.31-0ubuntu9.2, automatic), libctf-nobfd0:amd64 (2.34-6ubuntu1.1, automatic), libexpat1-dev:amd64 (2.2.9-1build1, automatic), cpp-9:amd64 (9.3.0-17ubuntu1~20.04, automatic), libalgorithm-diff-perl:amd64 (1.19.03-2, automatic), libalgorithm-merge-perl:amd64 (0.08-3, automatic), binutils:amd64 (2.34-6ubuntu1.1, automatic), cpp:amd64 (4:9.3.0-1ubuntu2, automatic), libitm1:amd64 (10.3.0-1ubuntu1~20.04, automatic), g++:amd64 (4:9.3.0-1ubuntu2, automatic), python3-pip:amd64 (20.0.2-5ubuntu1.5), python3-wheel:amd64 (0.34.2-1, automatic), gcc:amd64 (4:9.3.0-1ubuntu2, automa...

[Drew Freiberger (afreiberger)]

tl;dr python3-pip requires apt-get update to be run before install via hook due to dependencies that don't exist or are not mapped to package revs in the apt cache in the focal image from 2021-06-22. I'm not sure why this host failed to apt-update on deployment.

[Billy Olsen (billy-olsen)]

It would be interesting to see what the cloud-init-*.log files contained as the apt-update is typically performed by the cloud-init scripts when first loading. Additionally, the model-config value of enable-os-upgrade/enable-os-refresh-update are typically set to True to indicate whether the apt-update is run. While it is true that the install script does not perform an apt-update (and could do so), it is better to understand why the initial apt-update failed so we can resolve that issue as well.