While attempting to validate the openstack-telemetry bundle for focal-yoga I ran
juju run-action --wait ceilometer/0 ceilometer-upgrade
Doing so led to failure:
unit-ceilometer-0: UnitId: ceilometer/0 id: "10" message: 'ceilometer-upgrade resulted in an unexpected error: Command ''[''ceilometer-upgrade'', ''--debug'', ''--retry'', ''10'']'' returned non-zero exit status 1.' results: Stdout: "2022-05-16 22:06:54.760 45489 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42\e[00m\n2022-05-16 22:06:59.706 45489 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException: {\"message\": \"The server is currently unavailable. Please try again at a later time.<br /><br />\\nThe Keystone service is temporarily unavailable.\\n\\n\", \"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer Traceback (most recent call last):\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/bin/ceilometer-upgrade\", line 10, in <module>\n2022-05-16 22:06:59.706 45489 ERROR ceilometer sys.exit(upgrade())\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py\", line 49, in upgrade\n2022-05-16 22:06:59.706 45489 ERROR ceilometer tenacity.Retrying(\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 423, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer do = self.iter(retry_state=retry_state)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer \ File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 360, in iter\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return fut.result()\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\", line 437, in result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.__get_result()\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\", line 389, in __get_result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise self._exception\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 426, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer result = fn(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py\", line 266, in upgrade_resource_types\n2022-05-16 22:06:59.706 45489 ERROR ceilometer gnocchi.resource_type.get(name=name)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py\", line 43, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self._get(self.url + name,\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py\", line 37, in _get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.client.api.get(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer \ File \"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py\", line 395, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.request(url, 'GET', **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/client.py\", line 52, in request\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise exceptions.from_response(resp, method)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer gnocchiclient.exceptions.ClientException: {\"message\": \"The server is currently unavailable. Please try again at a later time.<br /><br />\\nThe Keystone service is temporarily unavailable.\\n\\n\", \"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer \e[00m\n" outcome: ceilometer-upgrade failed, see traceback. traceback: | Traceback (most recent call last): File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 669, in ceilometer_upgrade_helper ceilometer_upgrade(action=True) File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 691, in ceilometer_upgrade subprocess.check_call(cmd) File "/usr/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['ceilometer-upgrade', '--debug', '--retry', '10']' returned non-zero exit status 1. status: failed
There appears to be a Keystone issue. I verified that I could create and connect to an instance (via SSH).
A juju crashdump file is attached.
While attempting to validate the openstack-telemetry bundle for focal-yoga I ran
juju run-action --wait ceilometer/0 ceilometer-upgrade
Doing so led to failure:
unit-ceilometer-0: -upgrade' ', cmd.storage [-] Upgrading python3/ dist-packages/ ceilometer/ cmd/storage. py:42\e[ 00m\n2022- 05-16 exceptions. ClientException : \\n\\n\ ", \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer Traceback (most recent bin/ceilometer- upgrade\ ", \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer sys.exit( upgrade( ))\n2022- 05-16 lib/python3/ dist-packages/ ceilometer/ cmd/storage. py\", Retrying( \n2022- 05-16 lib/python3/ dist-packages/ tenacity/ __init_ _.py\", _\n2022- 05-16 22:06:59.706 45489 ERROR ceilometer do retry_state= retry_state) \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer lib/python3/ dist-packages/ tenacity/ __init_ _.py\", line 360, in n2022-05- 16 22:06:59.706 45489 ERROR ceilometer return fut.result( )\n2022- 05-16 lib/python3. 8/concurrent/ futures/ _base.py\ ", __get_result( )\n2022- 05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/ lib/python3. 8/concurrent/ futures/ _base.py\ ", n2022-05- 16 22:06:59.706 45489 ERROR ceilometer raise _exception\ n2022-05- 16 22:06:59.706 45489 ERROR ceilometer File \"/usr/ lib/python3/ dist-packages/ tenacity/ __init_ _.py\", _\n2022- 05-16 22:06:59.706 45489 ERROR ceilometer result \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer File "/usr/lib/ python3/ dist-packages/ ceilometer/ gnocchi_ client. py\", line 266, in resource_ types\n2022- 05-16 22:06:59.706 45489 ERROR ceilometer gnocchi. resource_ type.get( name=name) \n2022- 05-16 lib/python3/ dist-packages/ gnocchiclient/ v1/resource_ type.py\ ", lib/python3/ dist-packages/ gnocchiclient/ v1/base. py\", client. api.get( *args, **kwargs) \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer lib/python3/ dist-packages/ keystoneauth1/ adapter. py\", line 395, \n2022- 05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/ lib/python3/ dist-packages/ gnocchiclient/ client. py\", s.from_ response( resp, method)\n2022-05-16 22:06:59.706 45489 ERROR exceptions. ClientException : {\"message\": \"The server \\n\\n\ ", \"code\": \"503 Service le\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706 juju/agents/ unit-ceilometer -0/charm/ actions/ ceilometer_ utils.py" , line 669, in ceilometer_ upgrade_ helper
ceilometer_ upgrade( action= True) juju/agents/ unit-ceilometer -0/charm/ actions/ ceilometer_ utils.py" , line 691, in ceilometer_upgrade
subprocess. check_call( cmd) python3. 8/subprocess. py", line 364, in check_call ror(retcode, cmd) s.CalledProcess Error: Command '['ceilometer- upgrade' , '--debug', '--retry', '10']' returned non-zero exit status 1.
UnitId: ceilometer/0
id: "10"
message: 'ceilometer-upgrade resulted in an unexpected error: Command ''[''ceilometer
''--debug'', ''--retry'', ''10'']'' returned non-zero exit status 1.'
results:
Stdout: "2022-05-16 22:06:54.760 45489 DEBUG ceilometer.
Gnocchi resource types upgrade /usr/lib/
22:06:59.706 45489 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.
{\"message\": \"The server is currently unavailable. Please try again at a later
time.<br /><br />\\nThe Keystone service is temporarily unavailable.
\"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP
503)
call last):\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 10, in <module>
22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 49, in upgrade\n2022-05-16 22:06:59.706 45489 ERROR ceilometer tenacity.
22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 423, in __call_
= self.iter(
\ File \"/usr/
iter\
22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 437, in result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.
line 389, in __get_result\
self.
line 426, in __call_
= fn(*args, **kwargs)
\
upgrade_
22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 43, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self._get(self.url
+ name,\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/
line 37, in _get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.
\ File \"/usr/
in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.request(url,
'GET', **kwargs)
line 52, in request\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
exception
ceilometer gnocchiclient.
is currently unavailable. Please try again at a later time.<br /><br />\\nThe
Keystone service is temporarily unavailable.
Unavailab
45489 ERROR ceilometer \e[00m\n"
outcome: ceilometer-upgrade failed, see traceback.
traceback: |
Traceback (most recent call last):
File "/var/lib/
File "/var/lib/
File "/usr/lib/
raise CalledProcessEr
subproces
status: failed
There appears to be a Keystone issue. I verified that I could create and connect to an instance (via SSH).
A juju crashdump file is attached.