Cluster die after upgrade during shaker full L2 tests

[Sergey Galkin]

Steps to reproduce
1. Deploy ccp with configs from attached tarball on the 95 nodes
2. Try to run shaker (http://pyshaker.readthedocs.io/en/latest/) scenario openstack/full_l2 with versions.yaml-mitaka as versions.yaml
3. When heat stack was created change versions.yaml to versions.yaml-mitaka and execute 'ccp deploymen'

Cluster die after upgrade. heat-engine hangs on the cluster deletion
1. New instance creation return Timed out waiting for a reply to message ID 39530f79a3f3434d8473af5ef80fa702
2. Heat in horizon return Gateway Timeout
3. Instances deletion hangs
4. In the logs of nova-conductor appear lines 'AMQP server on rpc.ccp.svc.cluster.local:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: 49058' but curl rpc.ccp.svc.cluster.local:5672 is working from this pod

[Sergey Galkin (sgalkin)]

1. New instance creation return Timed out waiting for a reply to message ID 39530f79a3f3434d8473af5ef80fa702

[Sergey Galkin (sgalkin)]

2. Heat in horizon return Gateway Timeout

[Sergey Galkin (sgalkin)]

3. Instances deletion hangs

[Sergey Galkin (sgalkin)]

4. In the logs of nova-conductor appear lines 'AMQP server on rpc.ccp.svc.cluster.local:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: 49058' but curl rpc.ccp.svc.cluster.local:5672 is working from this pod

[Sergey Galkin (sgalkin)]

This is network hi load issue