Related to lp:1927277. Can be reproduced by running the series-upgrade tests. [1] Here is what happens:
1. nova-compute (principal) and ceilometer-agent (subordinate) are running on the same bionic machine.
2. A series upgrade of that machine is started. This runs the pre-series-upgrade hooks of both charms, effectively stopping both the nova-compute and the ceilometer-agent-compute services.
3. `juju upgrade-series <machine-number> complete` is run. This runs ceilometer-agent's post-series-upgrade hook, attempting to start the ceilometer-agent-compute service. This service however has a dependency to the nova-compute service, which is still stopped. This produces a hook error.
Zaza logs:
----------
2021-11-30 18:41:53 [INFO] About to upgrade leader of nova-compute: 15
2021-11-30 18:41:53 [INFO] About to series-upgrade (15)
2021-11-30 18:41:53 [INFO] About to call '['juju', 'run', '--machine=15', '--', 'echo \'DPkg::options { "--force-confdef"; "--force-confnew"; }\' | sudo tee /etc/apt/apt.conf.d/local']'
...
2021-11-30 18:42:02 [INFO] About to call '['juju', 'run', '--machine=15', '--', 'yes | sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" dist-upgrade']'
...
2021-11-30 18:42:16 [INFO] About to call '['juju', 'run', '--machine=15', '--timeout=120m', '--', 'yes | sudo DEBIAN_FRONTEND=noninteractive do-release-upgrade -f DistUpgradeViewNonInteractive']'
...
2021-11-30 19:00:33 [INFO] About to call '['juju', 'upgrade-series', '-m', 'zaza-97c5ac52fcdc', '15', 'complete']'
----------
ceilometer-agent unit log:
----------
2021-11-30 19:01:00 WARNING post-series-upgrade Removed /etc/systemd/system/memcached.service.
2021-11-30 19:01:01 WARNING post-series-upgrade Synchronizing state of memcached.service with SysV service script with /lib/systemd/systemd-sysv-install.
2021-11-30 19:01:01 WARNING post-series-upgrade Executing: /lib/systemd/systemd-sysv-install enable memcached
2021-11-30 19:01:02 WARNING post-series-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/memcached.service → /lib/systemd/system/memcached.service.
2021-11-30 19:01:02 DEBUG post-series-upgrade inactive
2021-11-30 19:01:02 WARNING post-series-upgrade Removed /etc/systemd/system/ceilometer-agent-compute.service.
2021-11-30 19:01:03 WARNING post-series-upgrade Synchronizing state of ceilometer-agent-compute.service with SysV service script with /lib/systemd/systemd-sysv-install.
2021-11-30 19:01:03 WARNING post-series-upgrade Executing: /lib/systemd/systemd-sysv-install enable ceilometer-agent-compute
2021-11-30 19:01:04 WARNING post-series-upgrade Created symlink /etc/systemd/system/multi-user.target.wants/ceilometer-agent-compute.service → /lib/systemd/system/ceilometer-agent-compute.service.
2021-11-30 19:01:04 DEBUG post-series-upgrade inactive
2021-11-30 19:01:04 WARNING post-series-upgrade Failed to start ceilometer-agent-compute.service: Unit nova-compute.service is masked.
2021-11-30 19:01:04 DEBUG post-series-upgrade active
2021-11-30 19:01:04 DEBUG post-series-upgrade inactive
2021-11-30 19:01:04 DEBUG jujuc server.go:211 running hook tool "status-set" for ceilometer-agent/1-post-series-upgrade-5326356148989293630
2021-11-30 19:01:04 WARNING post-series-upgrade Traceback (most recent call last):
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/post-series-upgrade", line 188, in <module>
2021-11-30 19:01:04 WARNING post-series-upgrade hooks.execute(sys.argv)
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/core/hookenv.py", line 962, in execute
2021-11-30 19:01:04 WARNING post-series-upgrade self._hooks[hook_name]()
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/post-series-upgrade", line 159, in post_series_upgrade
2021-11-30 19:01:04 WARNING post-series-upgrade series_upgrade_complete(
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/utils.py", line 2178, in series_upgrade_complete
2021-11-30 19:01:04 WARNING post-series-upgrade resume_unit_helper(configs)
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/ceilometer_utils.py", line 308, in resume_unit_helper
2021-11-30 19:01:04 WARNING post-series-upgrade _pause_resume_helper(resume_unit, configs)
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/hooks/ceilometer_utils.py", line 320, in _pause_resume_helper
2021-11-30 19:01:04 WARNING post-series-upgrade f(assess_status_func(configs),
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/juju/agents/unit-ceilometer-agent-1/charm/charmhelpers/contrib/openstack/utils.py", line 1738, in resume_unit
2021-11-30 19:01:04 WARNING post-series-upgrade raise Exception("Couldn't resume: {}".format("; ".join(messages)))
2021-11-30 19:01:04 WARNING post-series-upgrade Exception: Couldn't resume: ceilometer-agent-compute didn't resume cleanly.; Services not running that should be: ceilometer-agent-compute
----------
Related to lp:1927277. Can be reproduced by running the series-upgrade tests. [1] Here is what happens:
1. nova-compute (principal) and ceilometer-agent (subordinate) are running on the same bionic machine. agent-compute services. agent-compute service. This service however has a dependency to the nova-compute service, which is still stopped. This produces a hook error.
2. A series upgrade of that machine is started. This runs the pre-series-upgrade hooks of both charms, effectively stopping both the nova-compute and the ceilometer-
3. `juju upgrade-series <machine-number> complete` is run. This runs ceilometer-agent's post-series-upgrade hook, attempting to start the ceilometer-
Zaza logs: apt.conf. d/local' ]' FRONTEND= noninteractive apt-get --assume-yes -o "Dpkg:: Options: :=--force- confdef" -o "Dpkg:: Options: :=--force- confold" dist-upgrade']' FRONTEND= noninteractive do-release-upgrade -f DistUpgradeView NonInteractive' ]' cdc', '15', 'complete']'
----------
2021-11-30 18:41:53 [INFO] About to upgrade leader of nova-compute: 15
2021-11-30 18:41:53 [INFO] About to series-upgrade (15)
2021-11-30 18:41:53 [INFO] About to call '['juju', 'run', '--machine=15', '--', 'echo \'DPkg::options { "--force-confdef"; "--force-confnew"; }\' | sudo tee /etc/apt/
...
2021-11-30 18:42:02 [INFO] About to call '['juju', 'run', '--machine=15', '--', 'yes | sudo DEBIAN_
...
2021-11-30 18:42:16 [INFO] About to call '['juju', 'run', '--machine=15', '--timeout=120m', '--', 'yes | sudo DEBIAN_
...
2021-11-30 19:00:33 [INFO] About to call '['juju', 'upgrade-series', '-m', 'zaza-97c5ac52f
----------
ceilometer-agent unit log: system/ memcached. service. systemd- sysv-install. systemd- sysv-install enable memcached system/ multi-user. target. wants/memcached .service → /lib/systemd/ system/ memcached. service. system/ ceilometer- agent-compute. service. agent-compute. service with SysV service script with /lib/systemd/ systemd- sysv-install. systemd- sysv-install enable ceilometer- agent-compute system/ multi-user. target. wants/ceilomete r-agent- compute. service → /lib/systemd/ system/ ceilometer- agent-compute. service. agent-compute. service: Unit nova-compute. service is masked. agent/1- post-series- upgrade- 532635614898929 3630 juju/agents/ unit-ceilometer -agent- 1/charm/ hooks/post- series- upgrade" , line 188, in <module> sys.argv) juju/agents/ unit-ceilometer -agent- 1/charm/ charmhelpers/ core/hookenv. py", line 962, in execute hook_name] () juju/agents/ unit-ceilometer -agent- 1/charm/ hooks/post- series- upgrade" , line 159, in post_series_upgrade upgrade_ complete( juju/agents/ unit-ceilometer -agent- 1/charm/ charmhelpers/ contrib/ openstack/ utils.py" , line 2178, in series_ upgrade_ complete unit_helper( configs) juju/agents/ unit-ceilometer -agent- 1/charm/ hooks/ceilomete r_utils. py", line 308, in resume_unit_helper resume_ helper( resume_ unit, configs) juju/agents/ unit-ceilometer -agent- 1/charm/ hooks/ceilomete r_utils. py", line 320, in _pause_ resume_ helper status_ func(configs) , juju/agents/ unit-ceilometer -agent- 1/charm/ charmhelpers/ contrib/ openstack/ utils.py" , line 1738, in resume_unit agent-compute didn't resume cleanly.; Services not running that should be: ceilometer- agent-compute
----------
2021-11-30 19:01:00 WARNING post-series-upgrade Removed /etc/systemd/
2021-11-30 19:01:01 WARNING post-series-upgrade Synchronizing state of memcached.service with SysV service script with /lib/systemd/
2021-11-30 19:01:01 WARNING post-series-upgrade Executing: /lib/systemd/
2021-11-30 19:01:02 WARNING post-series-upgrade Created symlink /etc/systemd/
2021-11-30 19:01:02 DEBUG post-series-upgrade inactive
2021-11-30 19:01:02 WARNING post-series-upgrade Removed /etc/systemd/
2021-11-30 19:01:03 WARNING post-series-upgrade Synchronizing state of ceilometer-
2021-11-30 19:01:03 WARNING post-series-upgrade Executing: /lib/systemd/
2021-11-30 19:01:04 WARNING post-series-upgrade Created symlink /etc/systemd/
2021-11-30 19:01:04 DEBUG post-series-upgrade inactive
2021-11-30 19:01:04 WARNING post-series-upgrade Failed to start ceilometer-
2021-11-30 19:01:04 DEBUG post-series-upgrade active
2021-11-30 19:01:04 DEBUG post-series-upgrade inactive
2021-11-30 19:01:04 DEBUG jujuc server.go:211 running hook tool "status-set" for ceilometer-
2021-11-30 19:01:04 WARNING post-series-upgrade Traceback (most recent call last):
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade hooks.execute(
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade self._hooks[
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade series_
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade resume_
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade _pause_
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade f(assess_
2021-11-30 19:01:04 WARNING post-series-upgrade File "/var/lib/
2021-11-30 19:01:04 WARNING post-series-upgrade raise Exception("Couldn't resume: {}".format("; ".join(messages)))
2021-11-30 19:01:04 WARNING post-series-upgrade Exception: Couldn't resume: ceilometer-
----------
[1]: https:/ /github. com/openstack- charmers/ charmed- openstack- tester