Comment 11 for bug 1952282

While testing the quincy/edge/chrome0 channel, we noticed a similar issue except that the failed hook is "radosgw-dashboard-relation-changed" (we can open a new bug if you prefer).

In the logs we see a couple of ceph commands failing with exist status 5, ending with:
==============
Traceback (most recent call last):
  File "./src/charm.py", line 378, in _run_cmd
    output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-audit-api-log-payload', 'True']' returned non-zero exit status 5.
2023-06-10 02:37:29 ERROR unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Command failed: b"Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]\n"
Traceback (most recent call last):
  File "./src/charm.py", line 378, in _run_cmd
    output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'motd', 'clear']' returned non-zero exit status 5.
2023-06-10 02:37:29 DEBUG unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Attempting to collect TLS config from relation
2023-06-10 02:37:29 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Updating certificates in /etc/ssl/certs...
2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 0 added, 0 removed; done.
2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Running hooks in /etc/ca-certificates/update.d...
2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 done.
2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: ['ceph', 'dashboard', 'set-ssl-certificate', 'juju-f98180-0-lxd-1', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]
2023-06-10 02:37:31 WARNING unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
2023-06-10 02:37:31 ERROR unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 632, in <module>
    main(CephDashboardCharm)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/src/interface_dashboard.py", line 50, in on_changed
    self.on.mon_ready.emit()
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 316, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 784, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 857, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 432, in _configure_dashboard
    self._configure_tls()
  File "./src/charm.py", line 557, in _configure_tls
    ceph_utils.dashboard_set_ssl_certificate(
  File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/charms_ceph/utils.py", line 3599, in _dashboard_set_ssl_artifact
    subprocess.check_call(cmd)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-ssl-certificate', 'juju-f98180-0-lxd-1', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]' returned non-zero exit status 5.
==============

`ceph dashboard debug disable` is one of them, just like the original hook failure.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/db5586e7-b34e-4e8f-ab6a-09f6a98070f5/index.html