Ceph Dashboard leader stays blocked: No certificates found.

[Bas de Bruijne]

In testrun https://solutions.qa.canonical.com/testruns/testRun/7a30e5f6-573a-4fa1-bf87-a294b687c776 , the ceph-dashboard leader stays down:

---------------------------------------------------------------------
ceph-mon/0* active idle 0/lxd/1 10.246.166.195 Unit is ready and clustered
  ceph-dashboard/0* blocked idle 10.246.166.195 No certificates found. Please add a certifcates relation or provide via charm config
  filebeat/20 active idle 10.246.166.195 Filebeat ready.
  landscape-client/20 maintenance idle 10.246.166.195 Need computer-title and juju-info to proceed
  logrotated/14 active idle 10.246.166.195 Unit is ready.
  nrpe/19 active idle 10.246.166.195 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  telegraf/18 active idle 10.246.166.195 9103/tcp Monitoring ceph-mon/0 (source version/commit 26e531a)
---------------------------------------------------------------------

In the logs we see that ceph-dashboard can't find the keyrings:

---------------------------------------------------------------------
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.704+0000 7f3582445700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.704+0000 7f3582445700 -1 AuthRegistry(0x7f357c05ac38) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.720+0000 7f3582445700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.720+0000 7f3582445700 -1 AuthRegistry(0x7f357c060638) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.720+0000 7f3582445700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 2022-03-29T18:21:04.720+0000 7f3582445700 -1 AuthRegistry(0x7f3582444010) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.update-status logger.go:60 [errno 2] RADOS object not found (error connecting to the cluster)
2022-03-29 18:21:04 WARNING unit.ceph-dashboard/0.juju-log server.go:327 Failed to list ceph modules: Command '['ceph', 'mgr', 'module', 'ls']' returned non-zero exit status 1.
---------------------------------------------------------------------

We have had the same SKU pass, so the relations and configs are right. There may be a problem in the relation rendering.

Link to crashdumps:
https://oil-jenkins.canonical.com/artifacts/7a30e5f6-573a-4fa1-bf87-a294b687c776/index.html