This was seen here: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ironic/+/791209
Perhaps a conflict of primary and subordinate charms installing packages at the same time?
juju status:
neutron-api/0* blocked idle 17 172.16.5.26 9696/tcp Services not running that should be: neutron-server
neutron-api-mysql-router/0* active idle 172.16.5.26 Unit is ready
neutron-ironic-agent/0* error idle 172.16.5.26 hook failed: "neutron-plugin-api-subordinate-relation-changed"
neutron-ironic-agent juju log:
2021-06-05 06:08:25 INFO juju-log neutron-plugin-api-subordinate:19: Couldn't acquire DPKG lock. Will retry in 10 seconds
2021-06-05 06:08:35 WARNING neutron-plugin-api-subordinate-relation-changed E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 38856 (unattended-upgr)
2021-06-05 06:08:35 WARNING neutron-plugin-api-subordinate-relation-changed E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?
2021-06-05 06:08:35 DEBUG jujuc server.go:211 running hook tool "juju-log" for neutron-ironic-agent/0-neutron-plugin-api-subordinate-relation-changed-6145776232064214029
2021-06-05 06:08:35 ERROR juju-log neutron-plugin-api-subordinate:19: Hook error:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/usr/local/lib/python3.8/dist-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/usr/local/lib/python3.8/dist-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/usr/local/lib/python3.8/dist-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-neutron-ironic-agent-0/charm/reactive/neutron_ironic_handlers.py", line 41, in install_ironic
charm_class.install()
File "lib/charm/openstack/neutron_ironic.py", line 31, in install
self.configure_source()
File "/usr/local/lib/python3.8/dist-packages/charms_openstack/charm/core.py", line 807, in configure_source
fetch.add_source(source, key)
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 627, in add_source
fn(*m.groups())
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 723, in _add_cloud_distro_check
_add_cloud_pocket("{}-{}".format(cloud_archive_release, openstack_release))
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 680, in _add_cloud_pocket
apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 303, in apt_install
_run_apt_command(cmd, fatal, quiet=quiet)
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 804, in _run_apt_command
_run_with_retries(
File "/usr/local/lib/python3.8/dist-packages/charmhelpers/fetch/ubuntu.py", line 781, in _run_with_retries
result = subprocess.check_call(cmd, env=env, **kwargs)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option=Dpkg::Options::=--force-confold', 'install']' returned non-zero exit status 100.
> Perhaps a conflict of primary and subordinate charms installing packages at the same time?
In theory this can't (shouldn't!?) happen as the juju agent for the machine is supposed to serialise hooks on a machine, so that a subordinate hook "can't" be running at the same time as a principle hook. At least it used to be this way?
I wondered if this was unattended upgrades getting in the way, but maybe that's a red-herring.