Action `ceilometer-upgrade` does not work

[Peter Matulis]

While attempting to validate the openstack-telemetry bundle for focal-yoga I ran

juju run-action --wait ceilometer/0 ceilometer-upgrade

Doing so led to failure:

unit-ceilometer-0:
  UnitId: ceilometer/0
  id: "10"
  message: 'ceilometer-upgrade resulted in an unexpected error: Command ''[''ceilometer-upgrade'',
    ''--debug'', ''--retry'', ''10'']'' returned non-zero exit status 1.'
  results:
    Stdout: "2022-05-16 22:06:54.760 45489 DEBUG ceilometer.cmd.storage [-] Upgrading
      Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42\e[00m\n2022-05-16
      22:06:59.706 45489 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException:
      {\"message\": \"The server is currently unavailable. Please try again at a later
      time.<br /><br />\\nThe Keystone service is temporarily unavailable.\\n\\n\",
      \"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP
      503)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer Traceback (most recent
      call last):\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/bin/ceilometer-upgrade\",
      line 10, in <module>\n2022-05-16 22:06:59.706 45489 ERROR ceilometer sys.exit(upgrade())\n2022-05-16
      22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py\",
      line 49, in upgrade\n2022-05-16 22:06:59.706 45489 ERROR ceilometer tenacity.Retrying(\n2022-05-16
      22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
      line 423, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer do
      = self.iter(retry_state=retry_state)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
      \ File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 360, in
      iter\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return fut.result()\n2022-05-16
      22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
      line 437, in result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
      self.__get_result()\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
      line 389, in __get_result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
      self._exception\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
      line 426, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer result
      = fn(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File
      \"/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py\", line 266, in
      upgrade_resource_types\n2022-05-16 22:06:59.706 45489 ERROR ceilometer gnocchi.resource_type.get(name=name)\n2022-05-16
      22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py\",
      line 43, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self._get(self.url
      + name,\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py\",
      line 37, in _get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
      self.client.api.get(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
      \ File \"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py\", line 395,
      in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.request(url,
      'GET', **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/client.py\",
      line 52, in request\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
      exceptions.from_response(resp, method)\n2022-05-16 22:06:59.706 45489 ERROR
      ceilometer gnocchiclient.exceptions.ClientException: {\"message\": \"The server
      is currently unavailable. Please try again at a later time.<br /><br />\\nThe
      Keystone service is temporarily unavailable.\\n\\n\", \"code\": \"503 Service
      Unavailable\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706
      45489 ERROR ceilometer \e[00m\n"
    outcome: ceilometer-upgrade failed, see traceback.
    traceback: |
      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 669, in ceilometer_upgrade_helper
          ceilometer_upgrade(action=True)
        File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 691, in ceilometer_upgrade
          subprocess.check_call(cmd)
        File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['ceilometer-upgrade', '--debug', '--retry', '10']' returned non-zero exit status 1.
  status: failed

There appears to be a Keystone + SSL issue. I verified that I could create and connect to an instance (via SSH).

A juju crashdump file is attached.

[Bas de Bruijne (basdbruijne)]

We are seeing this with SQA in the jammy-yoga testing as well: https://solutions.qa.canonical.com/testruns/testRun/d90316be-ce72-4933-8eee-31d28f48600e

Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/d90316be-ce72-4933-8eee-31d28f48600e/index.html

[Bas de Bruijne (basdbruijne)]

As a workaround you can remove and re-add the gnocchi-ceilometer and gnocchi-keystone relations. I assume something went wrong with updating the gnocchi endpoint from keystone on the first try, but I can't find anything in the logs.