restarted cluster after poweroutage has issue selecting leader

[Amit Sehgal]

Hi Folks,

After power outage, i've restarted the openstack services. However ceph-mon and ceph-osd has issues coming up.

2017-07-17 16:14:55.566174 7fd6f0bff700 1 mon.juju-b49a67-1-lxd-0@0(electing).elector(23) init, last seen epoch 23
2017-07-17 16:14:56.326854 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:14:56.327104 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:14:57.431483 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:14:57.431694 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:14:58.535081 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:14:58.535282 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:14:59.639169 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:14:59.639401 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:00.567264 7fd6f0bff700 1 mon.juju-b49a67-1-lxd-0@0(electing).elector(23) init, last seen epoch 23
2017-07-17 16:15:00.746615 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:00.746909 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:01.851430 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:01.851645 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:02.954925 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:02.955163 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:04.060351 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:04.060591 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:05.165156 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:05.165399 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2017-07-17 16:15:05.568390 7fd6f0bff700 1 mon.juju-b49a67-1-lxd-0@0(electing).elector(23) init, last seen epoch 23
2017-07-17 16:15:06.269307 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2017-07-17 16:15:06.269540 7fd6f3b45700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished

[Xiaojun Liao (wwba)]

This may be have something to do with data integrity during power loss, do you have disabled all caches coupled with non-volatile stores?