Create an instance using the new security group in created subnet fails with error: no eligible l3 agent found.

[Tatyanka]

Steps:
Shut down primary controller for Neutron

Scenario:
1. Deploy environment with 3+ controllers and NeutronTUN or NeutronVLAN, all default storages, 2 compute, 1 cinder node
2. Shut down primary controller
3. Verify networks
4. Ensure that VIPs are moved to other controller
5. Ensure connectivity to outside world from VM
6. Run OSTF tests
7. Turn on primary controller
8. Wait 5-10 minutes
9. Verify networks
10. Run OSTF tests

Actual result:
OSTF test on instance creation fails : 6. Create an instance using the new security group
in created subnet.

In neutron-server:
neutron.db.l3_agentschedulers_db [req-38ff38a1-5a9d-4df3-bb5f-fda28be75881 - - - - -] Failed to reschedule router 7efea45e-3843-4c6e-9552-973f7b787a8a
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db Traceback (most recent call last):
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 137, in reschedule_routers_from_down_agents
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db self.reschedule_router(context, binding.router_id)
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 314, in reschedule_router
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db router_id=router_id)
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db RouterReschedulingFailed: Failed rescheduling router 7efea45e-3843-4c6e-9552-973f7b787a8a: no eligible l3 agent found.
2016-01-18 17:20:47.956 2793 ERROR neutron.db.l3_agentschedulers_db

in l3 - agent
http://paste.openstack.org/show/484183/

At the same time rabbit looks healthy (at least passed ostf test):
http://paste.openstack.org/show/484184/

crm status
http://paste.openstack.org/show/484187/

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

[Oleg Bondarev (obondarev)]

Logs show that agents were unable to report state due to MessagingTimeout from ~ 16.32 up to the end ~17.30 which points to problems with rabbit. We need an env with repro to be able to debug together with messaging folks. Marking as incomplete

[Tatyanka (tatyana-leontovich)]

try to reproduce on the 493 iso , and case passed, also seems that the same scenario was success on iso 478 , so move to invalid