ceph-osd error on mon-iscsi relation

[Michael Quiniola]

Versions:
ceph-osd/304
ceph-mon/49

Upon relating ceph-iscsi to ceph-mon, all OSD's error with:
```hook failed: "mon-relation-changed"```

Error in one of the OSDs:

2020-09-23 20:32:58 DEBUG mon-relation-changed admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
2020-09-23 20:32:58 DEBUG mon-relation-changed Traceback (most recent call last):
2020-09-23 20:32:58 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-30/charm/hooks/mon-relation-changed", line 887, in <module>
2020-09-23 20:32:58 DEBUG mon-relation-changed hooks.execute(sys.argv)
2020-09-23 20:32:58 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-30/charm/hooks/charmhelpers/core/hookenv.py", line 945, in execute
2020-09-23 20:32:58 DEBUG mon-relation-changed self._hooks[hook_name]()
2020-09-23 20:32:58 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-30/charm/hooks/mon-relation-changed", line 666, in mon_relation
2020-09-23 20:32:58 DEBUG mon-relation-changed ceph.apply_osd_settings(settings)
2020-09-23 20:32:58 DEBUG mon-relation-changed File "lib/charms_ceph/utils.py", line 3327, in apply_osd_settings
2020-09-23 20:32:58 DEBUG mon-relation-changed subprocess.check_output(cmd.split()).decode('UTF-8'))
2020-09-23 20:32:58 DEBUG mon-relation-changed File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
2020-09-23 20:32:58 DEBUG mon-relation-changed return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2020-09-23 20:32:58 DEBUG mon-relation-changed File "/usr/lib/python3.8/subprocess.py", line 512, in run
2020-09-23 20:32:58 DEBUG mon-relation-changed raise CalledProcessError(retcode, process.args,
2020-09-23 20:32:58 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['ceph', 'daemon', 'osd.3', 'config', '--format=json', 'get', 'osd_heartbeat_grace']' returned non-zero exit status 22.
2020-09-23 20:32:58 ERROR juju.worker.uniter.operation runhook.go:136 hook "mon-relation-changed" (via explicit, bespoke hook script) failed: exit status 1

[Michael Quiniola (qthepirate)]

Tried the offending command in terminal and got this:

admin_socket: exception getting command descriptions: [Errno 2] No such file or directory

[Michael Quiniola (qthepirate)]

ISSUE FOUND!

This issue is caused if there were previous OSD's on the host. Example:

I removed and deployed 2 osd's, so I removed ceph-1 and ceph-2, then added ceph-3 ceph-4.
In /var/lib/ceph/osd, folder for ceph-1 and ceph-2 are still there even after removal. When attaching the iscsi charm to the mon, when the osd changes kick in, it errors because a specific osd is no longer there but it thinks it is.

Workaround:

delete offending OSD folders from /var/lib/ceph/osd directory (assuming that osd is no longer being used and is no longer in crush map)