It's observed in the CI job too: https://review.opendev.org/c/openstack/charm-ceph-mon/+/913814/1#message-0ca38a491712c9c6811160afa15c12337aca3ece 2024-03-22 00:14:23.622503 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: 2024-03-22 00:14:20 ERROR unit.ceph-mon/1.juju-log server.go:316 mon:0: Uncaught exception while in charm code: 2024-03-22 00:14:23.622591 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: Traceback (most recent call last): 2024-03-22 00:14:23.622706 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/./src/charm.py", line 317, in 2024-03-22 00:14:23.622795 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: main(CephMonCharm) 2024-03-22 00:14:23.622909 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/main.py", line 436, in main 2024-03-22 00:14:23.623006 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: _emit_charm_event(charm, dispatcher.event_name) 2024-03-22 00:14:23.623125 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/main.py", line 144, in _emit_charm_event 2024-03-22 00:14:23.623214 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: event_to_emit.emit(*args, **kwargs) 2024-03-22 00:14:23.623309 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 351, in emit 2024-03-22 00:14:23.623396 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: framework._emit(event) 2024-03-22 00:14:23.623483 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 853, in _emit 2024-03-22 00:14:23.623570 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: self._reemit(event_path) 2024-03-22 00:14:23.623658 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 942, in _reemit 2024-03-22 00:14:23.623778 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: custom_handler(event) 2024-03-22 00:14:23.623879 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/./src/charm.py", line 132, in on_mon_relation 2024-03-22 00:14:23.623993 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if hooks.mon_relation(): 2024-03-22 00:14:23.624085 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 558, in mon_relation 2024-03-22 00:14:23.624172 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: notify_relations() 2024-03-22 00:14:23.624260 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 631, in notify_relations 2024-03-22 00:14:23.624361 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: notify_osds() 2024-03-22 00:14:23.624464 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 652, in notify_osds 2024-03-22 00:14:23.624554 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: osd_relation(relid=relid, unit=unit) 2024-03-22 00:14:23.624642 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 897, in osd_relation 2024-03-22 00:14:23.624729 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: execute_post_osd_upgrade_steps(ceph_osd_releases[0]) 2024-03-22 00:14:23.624830 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/utils.py", line 321, in execute_post_osd_upgrade_steps 2024-03-22 00:14:23.624934 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if (_all_ceph_versions_same() and 2024-03-22 00:14:23.625044 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/utils.py", line 352, in _all_ceph_versions_same 2024-03-22 00:14:23.625166 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if len(versions_dict['osd']) < 1: 2024-03-22 00:14:23.625262 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: KeyError: 'osd' 2024-03-22 00:14:23.625594 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: 2024-03-22 00:14:21 ERROR juju.worker.uniter.operation runhook.go:153 hook "mon-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1