ceph-osd is not able to restart after ceph cluster is crashed

[jin-Ho david shin]

I have installed Ceph using the latest Ceph helm chart (release 0.10) on two physical servers.

After rebooting the servers, the ceph-osd pods are not able to restart and giving the following error message

+ echo 'ceph-0: /usr/bin/ceph-osd --cluster ceph -f -i 0 --osd-journal /var/lib/ceph/osd/ceph-0//journal -k /var/lib/ceph/osd/ceph-0//keyring'
+ tee -a /etc/forego/ceph/Procfile
ceph-0: /usr/bin/ceph-osd --cluster ceph -f -i 0 --osd-journal /var/lib/ceph/osd/ceph-0//journal -k /var/lib/ceph/osd/ceph-0//keyring
+ log SUCCESS
+ '[' -z SUCCESS ']'
++ date '+%F %T'
2017-09-12 16:16:03 /entrypoint.sh: SUCCESS
+ TIMESTAMP='2017-09-12 16:16:03'
+ echo '2017-09-12 16:16:03 /entrypoint.sh: SUCCESS'
+ return 0
+ start_forego
+ exec /usr/local/bin/forego start -f /etc/forego/ceph/Procfile
forego | starting ceph-0.1 on port 5000
ceph-0.1 | starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0//journal
ceph-0.1 | 2017-09-12 16:16:03.948496 7f61a8e7d8c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
ceph-0.1 | 2017-09-12 16:16:06.287924 7f61a8e7d8c0 -1 osd.0 105 log_to_monitors {default=true}
ceph-0.1 | 2017-09-12 16:16:06.356920 7f61a8e7d8c0 -1 ** ERROR: osd init failed: (1) Operation not permitted

Could you help me to know how I could restart the pods?

[Alan Meadows (alan-meadows)]

This can sometimes occur when previous ceph cluster data is left behind in /var/lib/openstack-helm from previous instantiations of ceph. The authentication details from the prior the ceph installs can hang around preventing the osd from registering with the monitors.

If this is an empty cluster please try purging /var/lib/openstack-helm across all physical hosts participating in ceph.