Removing a ceph-remote-relation-departed relation makes the charm crash.
It appears that the charm tries to re-run setup during the departed hook; this fails as it tries to re-add pool mirrors.
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: tracer>
tracer: starting handler dispatch, 32 flags set
tracer: set flag ceph-local.available
tracer: set flag ceph-local.connected
tracer: set flag ceph-rbd-mirror-installed
tracer: set flag ceph-remote.available
tracer: set flag ceph-remote.connected
tracer: set flag charm.installed
tracer: set flag charms.openstack.do-default-charm.installed
tracer: set flag charms.openstack.do-default-config-rendered
tracer: set flag charms.openstack.do-default-update-status
tracer: set flag charms.openstack.do-default-upgrade-charm
tracer: set flag config.default.source
tracer: set flag config.default.use-syslog
tracer: set flag config.rendered
tracer: set flag config.set.source
tracer: set flag endpoint.ceph-local.changed
tracer: set flag endpoint.ceph-local.changed.broker_requests
tracer: set flag endpoint.ceph-local.changed.ceph-cluster-address
tracer: set flag endpoint.ceph-local.changed.egress-subnets
tracer: set flag endpoint.ceph-local.changed.ingress-address
tracer: set flag endpoint.ceph-local.changed.pools
tracer: set flag endpoint.ceph-local.changed.private-address
tracer: set flag endpoint.ceph-local.joined
tracer: set flag endpoint.ceph-remote.changed
tracer: set flag endpoint.ceph-remote.changed.broker-rsp-ceph-rbd-mirror-b-0
tracer: set flag endpoint.ceph-remote.changed.broker_requests
tracer: set flag endpoint.ceph-remote.changed.ceph-cluster-address
tracer: set flag endpoint.ceph-remote.changed.egress-subnets
tracer: set flag endpoint.ceph-remote.changed.ingress-address
tracer: set flag endpoint.ceph-remote.changed.pools
tracer: set flag endpoint.ceph-remote.changed.private-address
tracer: set flag endpoint.ceph-remote.departed
tracer: set flag leadership.is_leader
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: tracer: hooks phase, 0 handlers queued
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: tracer>
tracer: main dispatch loop, 8 handlers queued
tracer: ++ queue handler hooks/relations/ceph-rbd-mirror/requires.py:54:joined:ceph-local
tracer: ++ queue handler hooks/relations/ceph-rbd-mirror/requires.py:58:changed:ceph-local
tracer: ++ queue handler hooks/relations/ceph-rbd-mirror/requires.py:58:changed:ceph-remote
tracer: ++ queue handler hooks/relations/ceph-rbd-mirror/requires.py:73:broken:ceph-remote
tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:109:broken:certificates
tracer: ++ queue handler reactive/ceph_rbd_mirror_handlers.py:100:configure_pools
tracer: ++ queue handler reactive/ceph_rbd_mirror_handlers.py:63:render_stuff
tracer: ++ queue handler reactive/layer_openstack.py:168:default_config_rendered
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Invoking reactive handler: reactive/ceph_rbd_mirror_handlers.py:63:render_stuff
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Ceph endpoint "ceph-local" available, configuring keyring
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Pools: "{'cinder-ceph': {'applications': {'rbd': {}}, 'parameters': {'pg_num': '128', 'size': '3'}, 'quota': {}}, 'device_health_metrics': {'applications': {'mgr_devicehealth': {}}, 'parameters': {'pg_num': '1', 'size': '3'}, 'quota': {}}, 'glance': {'applications': {'rbd': {}}, 'parameters': {'pg_num': '4', 'size': '3'}, 'quota': {}}, 'zaza': {'applications': {'rbd': {}}, 'parameters': {'pg_num': '8', 'size': '3'}, 'quota': {}}}"
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 creating /var/lib/ceph/rbd-mirror/ceph.client.rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21.keyring
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 added entity client.rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21 auth(key=AQDlCu5j02UeBhAAXjy9Puu62KwcYuqGdekg5Q==)
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Ceph endpoint "{}" flagged available yet no key. Relation is probably departing.
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Invoking reactive handler: reactive/ceph_rbd_mirror_handlers.py:100:configure_pools
2023-02-16 11:33:28 INFO unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Enabling mirroring for pool "cinder-ceph"
2023-02-16 11:33:28 DEBUG unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 note: changing mirroring mode from image to pool
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 rbd: multiple RX peers are not currently supported
2023-02-16 11:33:29 ERROR unit.ceph-rbd-mirror-b/0.juju-log server.go:316 ceph-remote:27: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/charm/reactive/ceph_rbd_mirror_handlers.py", line 129, in configure_pools
charm_instance.mirror_pool_enable(pool, pool_mirroring_mode)
File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/charm/lib/charm/openstack/ceph_rbd_mirror.py", line 162, in mirror_pool_enable
subprocess.check_call(base_cmd + ['peer', 'add', pool,
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rbd', '--id', 'rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21', 'mirror', 'pool', 'peer', 'add', 'cinder-ceph', 'client.rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21@remote']' returned non-zero exit status 22.
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 Traceback (most recent call last):
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/charm/hooks/ceph-remote-relation-departed", line 22, in <module>
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 main()
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 bus.dispatch(restricted=restricted_mode)
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 _invoke(other_handlers)
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 handler.invoke()
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 self._action(*args)
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/charm/reactive/ceph_rbd_mirror_handlers.py", line 129, in configure_pools
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 charm_instance.mirror_pool_enable(pool, pool_mirroring_mode)
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/var/lib/juju/agents/unit-ceph-rbd-mirror-b-0/charm/lib/charm/openstack/ceph_rbd_mirror.py", line 162, in mirror_pool_enable
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 subprocess.check_call(base_cmd + ['peer', 'add', pool,
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 raise CalledProcessError(retcode, cmd)
2023-02-16 11:33:29 WARNING unit.ceph-rbd-mirror-b/0.ceph-remote-relation-departed logger.go:60 subprocess.CalledProcessError: Command '['rbd', '--id', 'rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21', 'mirror', 'pool', 'peer', 'add', 'cinder-ceph', 'client.rbd-mirror.juju-06fad1-zaza-6873e749c8ee-21@remote']' returned non-zero exit status 22.
2023-02-16 11:33:29 ERROR juju.worker.uniter.operation runhook.go:140 hook "ceph-remote-relation-departed" (via explicit, bespoke hook script) failed: exit status 1
CI artifacts:
https:/