"unexpected vif_type=binding_failed during 21h shaker tests run

[Sergey Galkin]

Steps to reproduce
1. Create env on neutron vlan
2. Run long (21 hours) shaker tests from attachment

udp_l3_north_south test failed with error

"Error while executing scenario: Failed to deploy Heat stack ddea56c4-f963-40d9-a627-8b65844284a4. Expected status COMPLETE, but got FAILED. Reason: Resource CREATE failed: ResourceInError: resources.shaker_rafmsk_master_2: Went to status ERROR due to \"Message: Build of instance f3469ef6-de36-417f-ac41-9c0cfcece5d2 was re-scheduled: Unexpected vif_type=binding_failed, Code: 500\"

Fuel version:
astute_sha: e24ca066bf6160bc1e419aaa5d486cad1aaa937d
auth_required: true
build_id: 2015-08-17_03-04-59
build_number: '182'
feature_groups:
- mirantis
fuel-agent_sha: 57145b1d8804389304cd04322ba0fb3dc9d30327
fuel-library_sha: 9de2625d26c3b88d22082baecd789b6bd5ddf3fa
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: 17786b86b78e5b66d2b1c15500186648df10c63d
fuelmain_sha: d8c726645be087bc67e2eeca134f0f9747cfeacd
nailgun_sha: 4710801a2f4a6d61d652f8f1e64215d9dde37d2e
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 4c74a60aa60c06c136d9197c7d09fa4f8c8e2863
release: '7.0'

[Eugene Nikanorov (enikanorov)]

Two compute nodes of the environment have OVS agent in 'hanging' state.
They don't write logs, don't sent heartbeats to server. However strace shows that agents are active - polling ovs and doing regular things.

[Sergey Galkin (sgalkin)]

snapshot http://mos-scale-share.mirantis.com/fuel-snapshot-2015-09-02_14-11-02.tar.xz

[Eugene Nikanorov (enikanorov)]

Several OVS agents became dead on the environment nearly at the same time.
Some of them stopped to send heartbeats, all of them stopped to write logs.

[Eugene Nikanorov (enikanorov)]

This was a moment of time when rabbitmq cluster have been partitioned and started to replicate queues.

[Eugene Nikanorov (enikanorov)]

Agents continued to work and write logs after rabbitmq restart.

[Eugene Nikanorov (enikanorov)]

This has something to do with oslo.messaging.

Reproduction should involve constant rabbitmq Up/Down and reassembling.

[Alexander Ignatov (aignatov)]

Need a repro one more time. Not clear what to do with this issue for now.

[Roman Podoliaka (rpodolyaka)]

No repro / feedback in a month. Closing as Invalid.