Split brain with etcd cluster

[Sergey Galkin]

Steps to reproduce

1. Checkout a6664a573d9cedc931e41db3b2f9126bf9e107b6 commit for fuel-ccp git repo
2. Try to deploy cluster with etcd cluster with this - http://paste.openstack.org/show/602474/ topology

Cluster successfully deployed but die after 2 days 'ccp status' and 'kubectl -n ccp get pods | grep etcd' out is http://paste.openstack.org/show/602475/

In 'kubectl logs --tail 10 database galera-checker'
Etcd is not ready: Key not found : /galera

Full log is http://paste.openstack.org/show/602476/

Twice run of 'curl -k http://etcd.ccp.svc.cluster.local:2379/v2/keys/?recursive=true | python -m json.tool | grep galera' return different results

http://paste.openstack.org/show/602478/

[Aleksander Mogylchenko (amogylchenko)]

It should be mentioned that cluster died after Docker restart on all nodes.

[Sergey Galkin (sgalkin)]

Yes. Docker was restarted node by node