Timed out waiting for a reply to message when primary controller gone away in HA

[Tatyanka]

{"build_id": "2014-07-04_13-44-50", "mirantis": "yes", "build_number": "97", "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f", "nailgun_sha": "d01b4efc0fc4af9d0e316b9dfc7974f16975f822", "production": "docker", "api": "1.0", "fuelmain_sha": "e312e03dbe29d3436958f7ac024402b1c468e2e4", "astute_sha": "644d279970df3daa5f5a2d2ccf8b4d22d53386ff", "release": "5.0.1", "fuellib_sha": "8a7d86a033b82520abe611bc2c286a10eae42d93"}

Steps to Reproduce:
1. Create cluster with neutron vlan
2. Add 3 nodes with controller role
3. Add 2 nodes with compute role
4. Deploy the cluster
5. Run network verification
6. Run OSTF
7. turn of primarry controller
8. leave env for 1h to be sure that time to recover was enaugh
9. Run OSTF
10. Go to the Horizon try to create instanse

Expected:
nova-manage marks offline (controller) with XXX, so only this infrastructure test is failed
All other ostf tests - passed
Intance in horizon created and Active

Actual:
Instance creation failed(btw inctance created before failover and get active state with success) with message:
nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Timed out waiting for a reply to message ID 41425d13985f49f09ac3e130f8a698c8

http://paste.openstack.org/show/85501/

See more details in attached logs(Btw a leave the env for additional 2 hours - and then re-run the checks - situation is the same)

[Tatyanka (tatyana-leontovich)]

The same problem after hard reboot of controller, so add one more snapshot here