Ceph monitors not responding (Error connecting to cluster: TimedOut)

[Paul Bourke]

Ceph mons are timing out, meaning the OSDs cannot bootstrap and cluster is unusable.

[root@storage01 ~]# docker logs bootstrap_osd_0
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
<snip>
INFO:__main__:Writing out command to execute
2016-09-30 10:06:57.882178 7fdb08f02700 0 monclient(hunting): authenticate timed out after 300
2016-09-30 10:06:57.882301 7fdb08f02700 0 librados: client.admin authentication error (110) Connection timed out
Error connecting to cluster: TimedOut

docker exec ceph_mon ceph -s

hangs.

[Paul Bourke (pauldbourke)]

Some logs from ceph_mon, seems to be continually trying to form a quorum:

2016-09-30 14:40:30.171053 7f5e2294e700 0 mon.192.168.7.253@2(electing).data_health(0) update_stats avail 90% total 34875 MB, used 1570 MB, avail 31435 MB
2016-09-30 14:40:36.153645 7f5e2214d700 0 log_channel(cluster) log [INF] : mon.192.168.7.253 calling new monitor election
2016-09-30 14:40:36.153786 7f5e2214d700 1 mon.192.168.7.253@2(electing).elector(175) init, last seen epoch 175
2016-09-30 14:40:51.187476 7f5e2214d700 0 log_channel(cluster) log [INF] : mon.192.168.7.253 calling new monitor election
2016-09-30 14:40:51.187574 7f5e2214d700 1 mon.192.168.7.253@2(electing).elector(177) init, last seen epoch 177
2016-09-30 14:41:06.223918 7f5e2214d700 0 log_channel(cluster) log [INF] : mon.192.168.7.253 calling new monitor election
2016-09-30 14:41:06.224077 7f5e2214d700 1 mon.192.168.7.253@2(electing).elector(179) init, last seen epoch 179
2016-09-30 14:41:21.264659 7f5e2214d700 0 log_channel(cluster) log [INF] : mon.192.168.7.253 calling new monitor election
2016-09-30 14:41:21.264792 7f5e2214d700 1 mon.192.168.7.253@2(electing).elector(181) init, last seen epoch 181
2016-09-30 14:41:30.171589 7f5e2294e700 0 mon.192.168.7.253@2(electing).data_health(0) update_stats avail 90% total 34875 MB, used 1570 MB, avail 31435 MB

[Jeffrey Zhang (jeffrey4l)]

Tried several times. I can not re-produce such issue.

[sean mooney (sean-k-mooney)]

i have been stuck with this issue for 3 weeks since before i went on vacation.
disabling dumb init was going to be my next debuging step so im glad that works.

ill test locally on my failing systems and comment back if patching it out fixes it for me.

[sean mooney (sean-k-mooney)]

by the way i was reproducing this on a ubuntu source build of master

[sean mooney (sean-k-mooney)]

in my case removing dumb-init does not help

diff --git a/docker/ceph/ceph-base/Dockerfile.j2 b/docker/ceph/ceph-base/Dockerfile.j2
index d3ebbe2..db91fe5 100644
--- a/docker/ceph/ceph-base/Dockerfile.j2
+++ b/docker/ceph/ceph-base/Dockerfile.j2
@@ -30,7 +30,9 @@ MAINTAINER {{ maintainer }}

 COPY extend_start.sh /usr/local/bin/kolla_extend_start
 RUN chmod 755 /usr/local/bin/kolla_extend_start \
- && usermod -a -G kolla ceph
+ && usermod -a -G kolla ceph ; \
+ sed -e "s?#!/usr/local/bin/dumb-init /bin/bash?#!/bin/bash?" -i /usr/local/bin/kolla_start
+

after rebuilding and redeploying i see the same behavior using the current head of master
f9f619d208413aee57309b7999227e05bcbf2e62

[sean mooney (sean-k-mooney)]

reducing for 6 mons to 3 resolved this for me with the dumb-init change.
i will check without the dumb-init change and see if it was jsut cause by
the fact i was deploying 6 mons tommorow.

[Jeffrey Zhang (jeffrey4l)]

i think the even number of ceph mon cause your issue.

[Paul Bourke (pauldbourke)]

@Jeffrey I have 3 mons, usually works. I'll retest tomorrow and confirm if this is still occurring as it sounds like its not consistent

[Paul Bourke (pauldbourke)]

Just deployed master copy of ceph (3 mons, 3 osds), no issues (git ce23dbe). I'm not 100% convinced this will not reappear but going to mark as invalid for now.