It's observed in the CI job too:
https://review.opendev.org/c/openstack/charm-ceph-mon/+/913814/1#message-0ca38a491712c9c6811160afa15c12337aca3ece
2024-03-22 00:14:23.622503 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: 2024-03-22 00:14:20 ERROR unit.ceph-mon/1.juju-log server.go:316 mon:0: Uncaught exception while in charm code: 2024-03-22 00:14:23.622591 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: Traceback (most recent call last): 2024-03-22 00:14:23.622706 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/./src/charm.py", line 317, in <module> 2024-03-22 00:14:23.622795 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: main(CephMonCharm) 2024-03-22 00:14:23.622909 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/main.py", line 436, in main 2024-03-22 00:14:23.623006 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: _emit_charm_event(charm, dispatcher.event_name) 2024-03-22 00:14:23.623125 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/main.py", line 144, in _emit_charm_event 2024-03-22 00:14:23.623214 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: event_to_emit.emit(*args, **kwargs) 2024-03-22 00:14:23.623309 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 351, in emit 2024-03-22 00:14:23.623396 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: framework._emit(event) 2024-03-22 00:14:23.623483 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 853, in _emit 2024-03-22 00:14:23.623570 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: self._reemit(event_path) 2024-03-22 00:14:23.623658 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/venv/ops/framework.py", line 942, in _reemit 2024-03-22 00:14:23.623778 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: custom_handler(event) 2024-03-22 00:14:23.623879 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/./src/charm.py", line 132, in on_mon_relation 2024-03-22 00:14:23.623993 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if hooks.mon_relation(): 2024-03-22 00:14:23.624085 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 558, in mon_relation 2024-03-22 00:14:23.624172 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: notify_relations() 2024-03-22 00:14:23.624260 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 631, in notify_relations 2024-03-22 00:14:23.624361 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: notify_osds() 2024-03-22 00:14:23.624464 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 652, in notify_osds 2024-03-22 00:14:23.624554 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: osd_relation(relid=relid, unit=unit) 2024-03-22 00:14:23.624642 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/ceph_hooks.py", line 897, in osd_relation 2024-03-22 00:14:23.624729 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: execute_post_osd_upgrade_steps(ceph_osd_releases[0]) 2024-03-22 00:14:23.624830 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/utils.py", line 321, in execute_post_osd_upgrade_steps 2024-03-22 00:14:23.624934 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if (_all_ceph_versions_same() and 2024-03-22 00:14:23.625044 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: File "/var/lib/juju/agents/unit-ceph-mon-1/charm/src/utils.py", line 352, in _all_ceph_versions_same 2024-03-22 00:14:23.625166 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: if len(versions_dict['osd']) < 1: 2024-03-22 00:14:23.625262 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: KeyError: 'osd' 2024-03-22 00:14:23.625594 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-mon-1.log: 2024-03-22 00:14:21 ERROR juju.worker.uniter.operation runhook.go:153 hook "mon-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
It's observed in the CI job too:
https:/ /review. opendev. org/c/openstack /charm- ceph-mon/ +/913814/ 1#message- 0ca38a491712c9c 6811160afa15c12 337aca3ece
2024-03-22 00:14:23.622503 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph- mon-1.log: 2024-03-22 00:14:20 ERROR unit.ceph- mon/1.juju- log server.go:316 mon:0: Uncaught exception while in charm code: mon-1.log: Traceback (most recent call last): mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ ./src/charm. py", line 317, in <module> mon-1.log: main(CephMonCharm) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ venv/ops/ main.py" , line 436, in main mon-1.log: _emit_charm_ event(charm, dispatcher. event_name) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ venv/ops/ main.py" , line 144, in _emit_charm_event mon-1.log: event_to_ emit.emit( *args, **kwargs) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ venv/ops/ framework. py", line 351, in emit mon-1.log: framework. _emit(event) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ venv/ops/ framework. py", line 853, in _emit mon-1.log: self._reemit( event_path) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ venv/ops/ framework. py", line 942, in _reemit mon-1.log: custom_ handler( event) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ ./src/charm. py", line 132, in on_mon_relation mon-1.log: if hooks.mon_ relation( ): mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/ceph_ hooks.py" , line 558, in mon_relation mon-1.log: notify_relations() mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/ceph_ hooks.py" , line 631, in notify_relations mon-1.log: notify_osds() mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/ceph_ hooks.py" , line 652, in notify_osds mon-1.log: osd_relation( relid=relid, unit=unit) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/ceph_ hooks.py" , line 897, in osd_relation mon-1.log: execute_ post_osd_ upgrade_ steps(ceph_ osd_releases[ 0]) mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/utils. py", line 321, in execute_ post_osd_ upgrade_ steps mon-1.log: if (_all_ceph_ versions_ same() and mon-1.log: File "/var/lib/ juju/agents/ unit-ceph- mon-1/charm/ src/utils. py", line 352, in _all_ceph_ versions_ same mon-1.log: if len(versions_ dict['osd' ]) < 1: mon-1.log: KeyError: 'osd' mon-1.log: 2024-03-22 00:14:21 ERROR juju.worker. uniter. operation runhook.go:153 hook "mon-relation- changed" (via hook dispatching script: dispatch) failed: exit status 1
2024-03-22 00:14:23.622591 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.622706 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.622795 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.622909 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623006 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623125 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623214 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623309 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623396 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623483 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623570 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623658 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623778 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623879 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.623993 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624085 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624172 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624260 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624361 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624464 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624554 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624642 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624729 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624830 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.624934 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.625044 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.625166 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.625262 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-
2024-03-22 00:14:23.625594 | focal-medium | 2024-03-22 00:14:23 [ERROR] unit-ceph-