ceilometer ceilometer-upgrade fail with Unhandled error
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Ceilometer Charm |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Fresh install of charm openstack from openstack-telemetry bundle with channel: 2023.1/stable, and series: jammy
After vault init, juju run ceilometer/leader ceilometer-upgrade reports unhandled error below:
$ juju run ceilometer/leader ceilometer-upgrade --debug
23:15:10 INFO juju.cmd supercommand.go:56 running juju [3.1.6 f6a66aa91eec620
23:15:10 DEBUG juju.cmd supercommand.go:57 args: []string{
23:15:10 INFO juju.juju api.go:86 connecting to API addresses: [192.168.
23:15:10 DEBUG juju.api apiclient.go:1171 successfully dialed "wss://
23:15:10 INFO juju.api apiclient.go:706 connection established to "wss://
23:15:10 INFO cmd common.go:377 Running operation 112 with 1 task
23:15:10 INFO cmd common.go:377 - task 113 on unit-ceilometer-2
23:15:10 INFO cmd common.go:377
23:15:10 INFO cmd common.go:377 Waiting for task 113...
outcome: ceilometer-upgrade failed, see traceback.
traceback: |
Traceback (most recent call last):
File "/var/lib/
ceilomete
File "/var/lib/
subproces
File "/usr/lib/
raise CalledProcessEr
subprocess.
2023-11-13 23:15:11.263 79134 DEBUG ceilometer.
2023-11-13 23:15:11.687 79134 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.
2023-11-13 23:15:11.687 79134 ERROR ceilometer Traceback (most recent call last):
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/bin/
2023-11-13 23:15:11.687 79134 ERROR ceilometer sys.exit(upgrade())
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer tenacity.Retrying(
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer do = self.iter(
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer return fut.result()
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.__get_result()
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer raise self._exception
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer result = fn(*args, **kwargs)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchi.
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self._get(self.url + name,
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.client.
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2023-11-13 23:15:11.687 79134 ERROR ceilometer File "/usr/lib/
2023-11-13 23:15:11.687 79134 ERROR ceilometer raise exceptions.
2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchiclient.
2023-11-13 23:15:11.687 79134 ERROR ceilometer
23:15:12 DEBUG juju.api monitor.go:35 RPC connection died
23:15:12 INFO cmd supercommand.go:535 command finished
with gnocchi unit ready:
$ juju status ceilometer gnocchi
Model Controller Cloud/Region Version SLA Timestamp
home-openstack juju-maas home-maas/default 3.1.6 unsupported 23:29:30Z
App Version Status Scale Charm Channel Rev Exposed Message
ceilometer 20.0.0 blocked 1 ceilometer 2023.1/stable 540 no Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
gnocchi 4.5.0 active 1 gnocchi 2023.1/stable 137 no Unit is ready
gnocchi-
Unit Workload Agent Machine Public address Ports Message
ceilometer/2* blocked idle 0/lxd/31 192.168.0.82 Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
gnocchi/3* active idle 0/lxd/33 192.168.0.78 8041/tcp Unit is ready
gnocchi-
Machine State Address Inst id Base AZ Message
0 started 192.168.0.4 opstk2464 ubuntu@22.04 default Deployed
0/lxd/31 started 192.168.0.82 juju-9d6257-
0/lxd/33 started 192.168.0.78 juju-9d6257-
Hi Manuel,
Thanks for filing this bug.
On Mon, 2023-11-13 at 23:38 +0000, Manuel Eurico Paula wrote: exceptions. ClientException :
>
> 2023-11-13 23:15:11.687 79134 ERROR ceilometer gnocchiclient.
> {"message": "The server is currently unavailable. Please try again at a later time.<br /><br
> />\nThe Keystone service is temporarily unavailable.\n\n", "code": "503 Service Unavailable",
> "title": "Service Unavailable"} (HTTP 503)
> 2023-11-13 23:15:11.687 79134 ERROR ceilometer
This exception suggests your Keystone deployment couldn't serve the request made by gnocchi. Can you
check the keystone logs during the same time window to correlate the underlying error?.
We have this page in the charm-guide that may be of help for you - /docs.openstack .org/charm- guide/latest/ community/ software- bug.html
https:/
Best,