ceilometer ceilometer-upgrade fail with Unhandled error

[Manuel Eurico Paula]

Fresh install of charm openstack from openstack-telemetry bundle with channel: 2023.1/stable, and series: jammy

After vault init, juju run ceilometer/leader ceilometer-upgrade reports unhandled error below:

$ juju run ceilometer/leader ceilometer-upgrade --debug
23:15:10 INFO juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620f5ac04a19d8c06bef03ae6228 gc go1.20.8]
23:15:10 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/24626/bin/juju", "run", "ceilometer/leader", "ceilometer-upgrade", "--debug"}
23:15:10 INFO juju.juju api.go:86 connecting to API addresses: [192.168.0.251:17070]
23:15:10 DEBUG juju.api apiclient.go:1171 successfully dialed "wss://192.168.0.251:17070/model/08a8862a-3c36-4c76-8668-ebfe6b9d6257/api"
23:15:10 INFO juju.api apiclient.go:706 connection established to "wss://192.168.0.251:17070/model/08a8862a-3c36-4c76-8668-ebfe6b9d6257/api"
23:15:10 INFO cmd common.go:377 Running operation 112 with 1 task
23:15:10 INFO cmd common.go:377 - task 113 on unit-ceilometer-2
23:15:10 INFO cmd common.go:377
23:15:10 INFO cmd common.go:377 Waiting for task 113...
outcome: ceilometer-upgrade failed, see traceback.
traceback: |
  Traceback (most recent call last):
    File "/var/lib/juju/agents/unit-ceilometer-2/charm/actions/ceilometer_utils.py", line 669, in ceilometer_upgrade_helper
      ceilometer_upgrade(action=True)
    File "/var/lib/juju/agents/unit-ceilometer-2/charm/actions/ceilometer_utils.py", line 691, in ceilometer_upgrade
      subprocess.check_call(cmd)
    File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['ceilometer-upgrade', '--debug', '--retry', '10']' returned non-zero exit status 1.

2023-11-13 23:15:11.263 79134 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42
2023-11-13 23:15:11.687 79134 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2023-11-13 23:15:11.687 79134 ERROR ceilometer Traceback (most recent call last):
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2023-11-13 23:15:11.687 79134 ERROR ceilometer sys.exit(upgrade())
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py", line 49, in upgrade
2023-11-13 23:15:11.687 79134 ERROR ceilometer tenacity.Retrying(
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 423, in __call__
2023-11-13 23:15:11.687 79134 ERROR ceilometer do = self.iter(retry_state=retry_state)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 360, in iter
2023-11-13 23:15:11.687 79134 ERROR ceilometer return fut.result()
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.__get_result()
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-11-13 23:15:11.687 79134 ERROR ceilometer raise self._exception
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 426, in __call__
2023-11-13 23:15:11.687 79134 ERROR ceilometer result = fn(*args, **kwargs)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py", line 260, in upgrade_resource_types
2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchi.resource_type.get(name=name)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py", line 43, in get
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self._get(self.url + name,
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
2023-11-13 23:15:11.687 79134 ERROR ceilometer raise exceptions.from_response(resp, method)
2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchiclient.exceptions.ClientException: {"message": "The server is currently unavailable. Please try again at a later time.<br /><br />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2023-11-13 23:15:11.687 79134 ERROR ceilometer
23:15:12 DEBUG juju.api monitor.go:35 RPC connection died
23:15:12 INFO cmd supercommand.go:535 command finished

with gnocchi unit ready:

$ juju status ceilometer gnocchi
Model Controller Cloud/Region Version SLA Timestamp
home-openstack juju-maas home-maas/default 3.1.6 unsupported 23:29:30Z

App Version Status Scale Charm Channel Rev Exposed Message
ceilometer 20.0.0 blocked 1 ceilometer 2023.1/stable 540 no Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
gnocchi 4.5.0 active 1 gnocchi 2023.1/stable 137 no Unit is ready
gnocchi-mysql-router 8.0.35 active 1 mysql-router 8.0/stable 111 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceilometer/2* blocked idle 0/lxd/31 192.168.0.82 Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
gnocchi/3* active idle 0/lxd/33 192.168.0.78 8041/tcp Unit is ready
  gnocchi-mysql-router/3* active idle 192.168.0.78 Unit is ready

Machine State Address Inst id Base AZ Message
0 started 192.168.0.4 opstk2464 ubuntu@22.04 default Deployed
0/lxd/31 started 192.168.0.82 juju-9d6257-0-lxd-31 ubuntu@22.04 default Container started
0/lxd/33 started 192.168.0.78 juju-9d6257-0-lxd-33 ubuntu@22.04 default Container started

[Felipe Reyes (freyes)]

Hi Manuel,

Thanks for filing this bug.

On Mon, 2023-11-13 at 23:38 +0000, Manuel Eurico Paula wrote:
>
> 2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchiclient.exceptions.ClientException:
> {"message": "The server is currently unavailable. Please try again at a later time.<br /><br
> />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable",
> "title": "Service Unavailable"} (HTTP 503)
> 2023-11-13 23:15:11.687 79134 ERROR ceilometer

This exception suggests your Keystone deployment couldn't serve the request made by gnocchi. Can you
check the keystone logs during the same time window to correlate the underlying error?.

We have this page in the charm-guide that may be of help for you -
https://docs.openstack.org/charm-guide/latest/community/software-bug.html

Best,

[Manuel Eurico Paula (manuel-paula)]

Hi Felipe,

As you suggested i grabbed the keystone log when ceilometer-upgrade is invoked:

root@juju-ace2d9-0-lxd-11:/var/log/keystone# cat keystone.log
(keystone.identity.core): 2023-11-25 08:32:43,195 WARNING Unable to locate domain config directory: /etc/keystone/domains
(keystone.common.password_hashing): 2023-11-25 08:32:43,211 WARNING Truncating password to algorithm specific maximum length 72 characters.

It seems is related to Bug: https://bugs.launchpad.net/charm-keystone/+bug/2012927

[Manuel Eurico Paula (manuel-paula)]

After refreshing ceilometer from revision 540 to 554 ceilometer-upgrade action worked.

App Version Status Scale Charm Channel Rev Exposed Message
ceilometer 20.0.0 active 1 ceilometer 2023.1/stable 554 no Unit is ready

[Alex Kavanagh (ajkavanagh)]

Judging by comment #5 this appears to be over-taken-by-events (OBE) and thus I'm marking it as invalid.