[Backport 1417267] neutron-ovs-agent on compute couldn't create interface

[Alexander Ignatov]

This is a backport of patch https://bugs.launchpad.net/neutron/+bug/1417267

================
Original description
================

Steps to reproduce:

1. Create env with two powerful compute nodes and start to provision instances.
2. After 130 instance one of the compute nodes starting failing provision because of unable to create interfaces. Here is trace from logs. http://paste.openstack.org/show/164308/
3. Restarting of agent restore functionality of compute node.
4. After some more VMs provisioned ovs-agent on another node stop working with the same symptoms.
5. Restarting of agent restore functionality of compute node.

[Dmitry Borodaenko (angdraug)]

Scalability to 100 nodes was only certified since 6.0, marked as Won't Fix for preceding releases.

[Dina Belova (dbelova)]

Dmitry, as you see the desciption you need only two powerful compute nodes, not 100 to reproduce the bug :) But anyway :)

[Eugene Nikanorov (enikanorov)]

Right now evidence show that the issue is outside of the neutron: ovs agent timeouts during different waiting of different rpc requests to neutron server.

Some of those timeouts lead to vif creation failure.

[Eugene Nikanorov (enikanorov)]

I'm moving state to incomplete and lowering priority so the bug is not weighting on mos-neutron team.

The repro cases that we've seen show that the reason was due to ovs agent was not able to receive information from neutron-server during rabbitmq outage that was caused by Fuel monitoring.
So right now https://bugs.launchpad.net/mos/+bug/1423404 is considered as a root cause of this issue.
There's not much that could be done on neutron (ovs agent) side.

We will reopen bug if this analysis is wrong.