In SolQA testrun - https://solutions.qa.canonical.com/testruns/7aa425f2-ab53-4a4e-9de5-40439041d1cb
```
ceph-mon/0 error idle 0/lxd/0 10.246.166.119 hook failed: "mon-relation-changed"
```
debug logs
2024-02-28 08:49:42 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running mon-relation-changed hook for ceph-mon/2
2024-02-28 08:49:42 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server {unix @/var/lib/juju/agents/unit-ceph-mon-1/agent.socket <nil>}
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: ops 2.9.0 up and running.
2024-02-28 08:49:42 WARNING unit.ceph-mon/1.juju-log server.go:325 mon:0: 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion.
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Emitting Juju event mon_relation_changed.
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/charm/ceph-mon ceph:ceph 555
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Writing file /var/lib/charm/ceph-mon/ceph.conf root:root 644
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/run/ceph ceph:ceph 755
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0 ceph:ceph 755
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 creating /var/lib/ceph/tmp/juju-31f10f-1-lxd-0.mon.keyring
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added entity mon. auth(key=AQCB895lE3iDKRAAWFYNdMO7MnctCwYQidpFUw==)
2024-02-28 08:49:42 DEBUG unit.ceph-mon/1.mon-relation-changed logger.go:60 added 1 caps to entity mon.
2024-02-28 08:49:42 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /<email address hidden> → /lib/systemd/system/ceph-mon@.service.
2024-02-28 08:49:42 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:45 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:48 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:52 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for quorum to be reached
2024-02-28 08:49:55 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Making dir /var/lib/ceph/mgr/ceph-juju-31f10f-1-lxd-0 ceph:ceph 555
2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Created symlink /<email address hidden> → /lib/systemd/system/ceph-mgr@.service.
2024-02-28 08:50:00 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: key 'autotune' doesn't exist
2024-02-28 08:50:01 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: Waiting for manager to be available
2024-02-28 08:50:06 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 set autotune
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 module 'pg_autoscaler' is already enabled (always-on)
2024-02-28 08:50:08 INFO unit.ceph-mon/1.juju-log server.go:325 mon:0: mon cluster in quorum - providing fsid & keys
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.697+0000 7fa885822640 -1 AuthRegistry(0x7fa88005fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:08 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:08.981+0000 7fb0820e5640 -1 AuthRegistry(0x7fb07c05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-upgrade in keyring
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-upgrade
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-upgrade', 'mon', 'allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del"']
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.269+0000 7fc249f51640 -1 AuthRegistry(0x7fc24405fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 2024-02-28T08:50:09.557+0000 7f17f0f1e640 -1 AuthRegistry(0x7f17ec05fd00) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
2024-02-28 08:50:09 WARNING unit.ceph-mon/1.mon-relation-changed logger.go:60 Error ENOENT: failed to find client.osd-removal in keyring
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Calling check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-juju-31f10f-1-lxd-0/keyring', 'auth', 'get-or-create', 'client.osd-removal', 'mgr', 'allow *', 'mon', 'allow r; allow command "osd crush reweight"; allow command "osd purge"; allow command "osd destroy"']0000 7fe3ef8e9640 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
2024-02-28 08:50:09 DEBUG unit.ceph-mon/1.juju-log server.go:325 mon:0: Creating new key for osd-removal
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/7aa425f2-ab53-4a4e-9de5-40439041d1cb/index.html
