Removing p-c-e application fails: hook ceph-relation-departed failed

[Vladimir Grevtsev]

focal/ussuri

unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: tracer: hooks phase, 0 handlers queued
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++ queue handler hooks/relations/ceph-client/requires.py:28:changed:ceph
tracer: ++ queue handler hooks/relations/ceph-client/requires.py:32:departed:ceph
tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:ceph-exporter
tracer: ++ queue handler reactive/prometheus_ceph_exporter.py:195:mon_relation_changed
tracer: ++ queue handler reactive/prometheus_ceph_exporter.py:227:remove_nrpe_check
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: reactive/prometheus_ceph_exporter.py:195:mon_relation_changed
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "status-set" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: tracer>
tracer: cleared flag exporter.started
tracer: ++ queue handler reactive/prometheus_ceph_exporter.py:103:configure_exporter
tracer: -- dequeue handler reactive/prometheus_ceph_exporter.py:195:mon_relation_changed
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: reactive/prometheus_ceph_exporter.py:227:remove_nrpe_check
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "relation-ids" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Check command not found: check_disk
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: hooks/relations/ceph-client/requires.py:28:changed:ceph
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: hooks/relations/ceph-client/requires.py:32:departed:ceph
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:ceph-exporter
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Invoking reactive handler: reactive/prometheus_ceph_exporter.py:103:configure_exporter
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: Changing permissions on existing content: 33060 -> 292
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: Changing permissions on existing content: 33060 -> 292
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 DEBUG unit.p-c-e/0.juju-log ceph:7: Changing permissions on existing content: 33060 -> 292
unit-p-c-e-0: 12:08:44 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:44 INFO unit.p-c-e/0.juju-log ceph:7: Starting snap.prometheus-ceph-exporter.ceph-exporter...
unit-p-c-e-0: 12:08:54 DEBUG unit.p-c-e/0.ceph-relation-departed failed
unit-p-c-e-0: 12:08:54 DEBUG jujuc running hook tool "status-set" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:54 DEBUG jujuc running hook tool "juju-log" for p-c-e/0-ceph-relation-departed-3163258924985288696
unit-p-c-e-0: 12:08:54 ERROR unit.p-c-e/0.juju-log ceph:7: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-p-c-e-0/charm/reactive/prometheus_ceph_exporter.py", line 144, in configure_exporter
    raise ServiceError("Service didn't start: {}".format(SVC_NAME))
reactive.prometheus_ceph_exporter.ServiceError: Service didn't start: snap.prometheus-ceph-exporter.ceph-exporter

unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed Traceback (most recent call last):
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/charm/hooks/ceph-relation-departed", line 22, in <module>
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed main()
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed bus.dispatch(restricted=restricted_mode)
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed _invoke(other_handlers)
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed handler.invoke()
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed self._action(*args)
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed File "/var/lib/juju/agents/unit-p-c-e-0/charm/reactive/prometheus_ceph_exporter.py", line 144, in configure_exporter
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed raise ServiceError("Service didn't start: {}".format(SVC_NAME))
unit-p-c-e-0: 12:08:54 WARNING unit.p-c-e/0.ceph-relation-departed reactive.prometheus_ceph_exporter.ServiceError: Service didn't start: snap.prometheus-ceph-exporter.ceph-exporter
unit-p-c-e-0: 12:08:54 ERROR juju.worker.uniter.operation hook "ceph-relation-departed" (via explicit, bespoke hook script) failed: exit status 1
unit-p-c-e-0: 12:08:54 DEBUG juju.machinelock machine lock released for p-c-e/0 uniter (run relation-departed (7; unit: ceph-mon/2, departee: p-c-e/0) hook)
unit-p-c-e-0: 12:08:54 DEBUG juju.worker.uniter.operation lock released for p-c-e/0
unit-p-c-e-0: 12:08:54 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-p-c-e-0: 12:08:54 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "ceph-relation-departed"

[Paul Goins (vultaire)]

Confirmed; I just hit this on a cloud myself.

[Paul Goins (vultaire)]

This may not be the cleanest workaround, but if you edit reactive/prometheus_ceph_exporter.py and comment out the "raise ServiceError" line, and the final line where it sets the flag indicating filebeat was started, that seems to be enough to get the charm to work through things and finally remove itself.

[Gábor Mészáros (gabor.meszaros)]

I got hit by this bug on a deployment too.

[Eric Chen (eric-chen)]

This charm is no longer being actively maintained. Please consider using the new Canonical Observability Stack instead. (https://charmhub.io/topics/canonical-observability-stack)