[library] Deployment of new controllers causes services downtime

[Artem Panchenko]

Fuel version info: http://paste.openstack.org/show/112149/

Steps to reproduce:

1. Create new cluster (Ubuntu, HA, NovaFlatDHCP, Ceph for images, Sahara, Ceilometer)
2. Add 3 controller+mongo, 4 compute and 4 cinder+ceph nodes (I tested on hardware)
3. Deploy changes. Run long-time (at least concurrency * 100 requests) Rally benchmark test on deployed cloud. I used concurrency = 80, but it could be smaller or greater depending on hardware specifications (cluster should be able to handle all incoming requests without overload/errors).
4. During performance testing add 2 new controllers to the environment and deploy changes

Expected result:

- deployment is successful, Rally test ends with 100% success (all requests to nova for create/delete instance were handled)

Actual result:

- deployment is successful, but Rally got a lot of 504/502 errors from HAProxy (Nova-api) and test was aborted:

http://paste.openstack.org/show/112154/

As you can see during few minutes nova-api was unreachable and most of requests from Rally failed. Here you can find the part of syslog on primary controller (node-18):

http://paste.openstack.org/show/112167/

And the part of HAProxy stats:

http://paste.openstack.org/show/112166/

Link to the diagnostic snapshot: https://yadi.sk/d/8yQXZY7EbWrmQ

[Irina Povolotskaya (ipovolotskaya)]

Should this be included into Release notes?
Is there any workaround?

[Dmitry Borodaenko (angdraug)]

This is a Medium priority bug so it doesn't qualify for release notes. The workaround is to re-run the failed API calls after controller deployment is finished.

[Vladimir Kuklin (vkuklin)]

This bug is one year old - there has been a lot of fixes since then. Marking as incomplete and asking qa guys to recheck this behaviour.

[Timur Nurlygayanov (tnurlygayanov)]

This bug was in incomplete state more than 1 month, status changed to Invalid.
Please, change the status to actual value if this bug is actual for now and add some comments.