Unit cannot be removed: hook failed: "ceph-mds-relation-departed"

[Vladimir Grevtsev]

focal/ussuri, charm cs:ceph-fs-29

App Version Status Scale Charm Store Rev OS Notes
ceph-fs 12.2.13 error 3 ceph-fs jujucharms 29 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-fs/0* error idle 3/lxd/7 172.16.151.127 hook failed: "ceph-mds-relation-departed"
ceph-fs/1 error idle 4/lxd/9 172.16.151.47 hook failed: "ceph-mds-relation-departed"
ceph-fs/2 error idle 5/lxd/8 172.16.151.141 hook failed: "ceph-mds-relation-departed"

steps to reproduce:

1. juju deploy cs: ceph-fs -n 3
2. juju add-relation ceph-fs:msd ceph-mon:mds
3. wait, until units will settle
4. juju remove-application ceph-fs

unit-ceph-fs-0: 10:19:39 INFO unit.ceph-fs/0.juju-log ceph-mds:368: Reactive main running for hook ceph-mds-relation-departed
unit-ceph-fs-0: 10:19:39 INFO unit.ceph-fs/0.juju-log ceph-mds:368: Invoking reactive handler: reactive/ceph_fs.py:39:config_changed
unit-ceph-fs-0: 10:19:39 ERROR unit.ceph-fs/0.juju-log ceph-mds:368: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ceph-fs-0/charm/reactive/ceph_fs.py", line 44, in config_changed
    cephfs_charm.configure_ceph_keyring(ceph_mds.mds_key())
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms_openstack/plugins/classes.py", line 393, in configure_ceph_keyring
    key, cluster_name=cluster_name)
  File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms_openstack/plugins/classes.py", line 192, in configure_ceph_keyring
    subprocess.check_call(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not NoneType

unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed Traceback (most recent call last):
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/charm/hooks/ceph-mds-relation-departed", line 22, in <module>
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed main()
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed bus.dispatch(restricted=restricted_mode)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed _invoke(other_handlers)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed handler.invoke()
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed self._action(*args)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/charm/reactive/ceph_fs.py", line 44, in config_changed
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed cephfs_charm.configure_ceph_keyring(ceph_mds.mds_key())
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms_openstack/plugins/classes.py", line 393, in configure_ceph_keyring
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed key, cluster_name=cluster_name)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/var/lib/juju/agents/unit-ceph-fs-0/.venv/lib/python3.6/site-packages/charms_openstack/plugins/classes.py", line 192, in configure_ceph_keyring
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed subprocess.check_call(cmd)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/usr/lib/python3.6/subprocess.py", line 306, in check_call
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed retcode = call(*popenargs, **kwargs)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/usr/lib/python3.6/subprocess.py", line 287, in call
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed with Popen(*popenargs, **kwargs) as p:
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed restore_signals, start_new_session)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed restore_signals, start_new_session, preexec_fn)
unit-ceph-fs-0: 10:19:39 WARNING unit.ceph-fs/0.ceph-mds-relation-departed TypeError: expected str, bytes or os.PathLike object, not NoneType
unit-ceph-fs-0: 10:19:40 ERROR juju.worker.uniter.operation hook "ceph-mds-relation-departed" (via explicit, bespoke hook script) failed: exit status 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-fs/+/815124

[Adam Dyess (addyess)]

Just providing another stack trace if necessary. it seems like there's a patch available for this bug which is not yet merged