[Kubernetes] cluster deployment fail

[Alex Kholkin]

Bug description:
Currently we have some problems with deploying of the kubernetes cluster. Sometimes deployment fails with the same error like in Actual results. It is not depend in some particular image(ubuntu or debian). The scenario down below shows the case when one of the env's deployment will fail for sure. Deployment of the single env with kubernetes cluster have the such problems sometimes.

Steps to reproduce:
1) Deploy devstack from master
2) Import Kubernetes cluster from a.o.o
3) go to Applications > Catalog > Environments
4) create 2 envs. Add Kubernetes cluster to both of them. One deploy with ubuntu14.04-x64-kubernetes.qcow2 and another one with debian8-x64-kubernetes.qcow2
5) deploy envs

Expected results:
both envs deployed successful

Actual results:
one of the env fail with error:

2016-08-16 12:00:59 — [murano.engine.system.agent.AgentException]: {'errorCode': 100, 'message': u'Script setup returned error code', 'extra': None, 'details': {u'stdout': u'Adding member kube-2 to etcd cluster\ncluster may be unhealthy: failed to list members\nMember kube-2 not added\nmember 607f0f2af09d2fbf is healthy: got healthy result from http://10.0.95.10:4001\ncluster is healthy\nMember kube-2 has been added\n\nAdding member kube-3 to etcd cluster\nmember 607f0f2af09d2fbf is healthy: got healthy result from http://10.0.95.10:4001\nmember fb561de2b41d86de is healthy: got healthy result from http://10.0.95.13:4001\ncluster is healthy\nMember kube-3 has been added\n\nAdding member gateway-1 to etcd cluster\nmember 607f0f2af09d2fbf is healthy: got healthy result from http://10.0.95.10:4001\nmember 98b9eb6df8b5a7f2 is healthy: got healthy result from http://10.0.95.5:4001\nmember fb561de2b41d86de is healthy: got healthy result from http://10.0.95.13:4001\ncluster is healthy\nMember gateway-1 not added. Reason: etcdserver: peerURL exists', u'stderr': None, u'exitCode': 1}, 'time': u'2016-08-16 11:59:27.414875'}

Reproducibility:
Always

Additional information:
Part of the murano-engine.log
https://paste.mirantis.net/show/2542/

[Dmytro Dovbii (ddovbii)]

etcd issues usually appear when your lab is slow. currently we are working on fix which makes etcd more stable on slow envs too

[Victor Ryzhenkin (vryzhenkin)]

@Dmytro, we have a lab with 24 VCPU, 64 GM RAM and SSD drives. Looks like this lab is not slow.

[Dmytro Dovbii (ddovbii)]

should be fixed by https://review.openstack.org/#/c/355796/
new images are also updated on a.o.o

could you please reverify?

[Alex Kholkin (akholkin)]

I made the same steps and got successful deployment.