As mentioned in #3 I am able to consistently reproduce this by running our automated OpenStack upgrade tests. [1]
Here is what is happening:
1. We pause the principle unit (nova-compute).
2. We perform an upgrade of all packages on the unit, including ceilometer-agent-compute. When upgrading this package, it wants to restart the service but the service refuses to start because nova-compute.service is masked.
3. We resume the principle unit (nova-compute), which restarts nova-compute.service but ceilometer-agent-compute.service is left behind, stopped.
Working on a solution.
unit-nova-compute-0.log:2021-08-17 11:41:59 DEBUG openstack-upgrade alembic ceilometer-agent-compute ceilometer-common keystone-common
unit-nova-compute-0.log:2021-08-17 11:41:59 DEBUG openstack-upgrade python3-ceilometer python3-cinderclient python3-cliff python3-cryptography
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:53 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 ceilometer-common all 1:15.0.0-0ubuntu2~cloud0 [27.0 kB]
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:54 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 ceilometer-agent-compute all 1:15.0.0-0ubuntu2~cloud0 [16.9 kB]
unit-nova-compute-0.log:2021-08-17 11:42:01 DEBUG openstack-upgrade Get:55 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 python3-ceilometer all 1:15.0.0-0ubuntu2~cloud0 [212 kB]
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../49-ceilometer-common_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer-common (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../50-ceilometer-agent-compute_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer-agent-compute (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../51-python3-ceilometer_1%3a15.0.0-0ubuntu2~cloud0_all.deb ...
unit-nova-compute-0.log:2021-08-17 11:42:15 DEBUG openstack-upgrade Unpacking python3-ceilometer (1:15.0.0-0ubuntu2~cloud0) over (1:14.0.0-0ubuntu0.20.04.3) ...
unit-nova-compute-0.log:2021-08-17 11:42:34 DEBUG openstack-upgrade Setting up ceilometer-common (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:42:34 DEBUG openstack-upgrade Configuration file '/etc/ceilometer/ceilometer.conf'
unit-nova-compute-0.log:2021-08-17 11:42:57 DEBUG openstack-upgrade Setting up python3-ceilometer (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:43:00 DEBUG openstack-upgrade Setting up ceilometer-agent-compute (1:15.0.0-0ubuntu2~cloud0) ...
unit-nova-compute-0.log:2021-08-17 11:43:00 DEBUG openstack-upgrade Installing new version of config file /etc/init.d/ceilometer-agent-compute ...
unit-nova-compute-0.log:2021-08-17 11:43:02 DEBUG openstack-upgrade Failed to restart ceilometer-agent-compute.service: Unit nova-compute.service is masked.
As mentioned in #3 I am able to consistently reproduce this by running our automated OpenStack upgrade tests. [1]
Here is what is happening:
1. We pause the principle unit (nova-compute). agent-compute. When upgrading this package, it wants to restart the service but the service refuses to start because nova-compute. service is masked. service but ceilometer- agent-compute. service is left behind, stopped.
2. We perform an upgrade of all packages on the unit, including ceilometer-
3. We resume the principle unit (nova-compute), which restarts nova-compute.
Working on a solution.
unit-nova- compute- 0.log:2021- 08-17 11:41:59 DEBUG openstack-upgrade alembic ceilometer- agent-compute ceilometer-common keystone-common compute- 0.log:2021- 08-17 11:41:59 DEBUG openstack-upgrade python3-ceilometer python3- cinderclient python3-cliff python3- cryptography compute- 0.log:2021- 08-17 11:42:01 DEBUG openstack-upgrade Get:53 http:// ubuntu- cloud.archive. canonical. com/ubuntu focal-updates/ victoria/ main amd64 ceilometer-common all 1:15.0. 0-0ubuntu2~ cloud0 [27.0 kB] compute- 0.log:2021- 08-17 11:42:01 DEBUG openstack-upgrade Get:54 http:// ubuntu- cloud.archive. canonical. com/ubuntu focal-updates/ victoria/ main amd64 ceilometer- agent-compute all 1:15.0. 0-0ubuntu2~ cloud0 [16.9 kB] compute- 0.log:2021- 08-17 11:42:01 DEBUG openstack-upgrade Get:55 http:// ubuntu- cloud.archive. canonical. com/ubuntu focal-updates/ victoria/ main amd64 python3-ceilometer all 1:15.0. 0-0ubuntu2~ cloud0 [212 kB] compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../49- ceilometer- common_ 1%3a15. 0.0-0ubuntu2~ cloud0_ all.deb ... compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer-common (1:15.0. 0-0ubuntu2~ cloud0) over (1:14.0. 0-0ubuntu0. 20.04.3) ... compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../50- ceilometer- agent-compute_ 1%3a15. 0.0-0ubuntu2~ cloud0_ all.deb ... compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Unpacking ceilometer- agent-compute (1:15.0. 0-0ubuntu2~ cloud0) over (1:14.0. 0-0ubuntu0. 20.04.3) ... compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Preparing to unpack .../51- python3- ceilometer_ 1%3a15. 0.0-0ubuntu2~ cloud0_ all.deb ... compute- 0.log:2021- 08-17 11:42:15 DEBUG openstack-upgrade Unpacking python3-ceilometer (1:15.0. 0-0ubuntu2~ cloud0) over (1:14.0. 0-0ubuntu0. 20.04.3) ... compute- 0.log:2021- 08-17 11:42:34 DEBUG openstack-upgrade Setting up ceilometer-common (1:15.0. 0-0ubuntu2~ cloud0) ... compute- 0.log:2021- 08-17 11:42:34 DEBUG openstack-upgrade Configuration file '/etc/ceilomete r/ceilometer. conf' compute- 0.log:2021- 08-17 11:42:57 DEBUG openstack-upgrade Setting up python3-ceilometer (1:15.0. 0-0ubuntu2~ cloud0) ... compute- 0.log:2021- 08-17 11:43:00 DEBUG openstack-upgrade Setting up ceilometer- agent-compute (1:15.0. 0-0ubuntu2~ cloud0) ... compute- 0.log:2021- 08-17 11:43:00 DEBUG openstack-upgrade Installing new version of config file /etc/init. d/ceilometer- agent-compute ... compute- 0.log:2021- 08-17 11:43:02 DEBUG openstack-upgrade Failed to restart ceilometer- agent-compute. service: Unit nova-compute. service is masked.
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
unit-nova-
[1] https:/ /github. com/openstack- charmers/ charmed- openstack- tester/ blob/master/ tests/openstack -upgrade/ tests/tests. yaml