There's a race condition between the time compute deletes a neutron port during destroy instance processing and the time the ovs_neutron_agent rpc_loop notices the ovs port removed and tries to update the associated neutron port to set the state to DOWN.
In our scenario, controller node is separate from the host, so these calls come over rest or rpc.
It appears that normally the ovs_neutron_agent wins and the rpc call for the update of the neutron port happens before compute does the rest api delete call. However, once in a while, compute's delete gets in first and deletes the port before ovs agent tries to update. In this case, the update fails, the failure is reported back via rpc, and the ovs agent then does a full resync on the next iteration in rpc_loop.
In a large scale environment, this is a problem because that resync can take a very long time due to a very large number of ports to reprocess. And while that single iteration is occurring (I have seen it take 10 minutes), new deploys start failing because the vif plug event timeout happens (since the agent will not process the port created for the plug until next iteration, which could be 10 minutes away, at which point the deploy has failed and cleaned up the port).
I think the fix for this, which I will create a patch for, is to not have treat_devices_removed part of the decision to resync. If we're removing a port, why do we care that it failed to find the neutron port? Not sure if there are other considerations to think about....
One intent that i can think of is w.r.t invalidating the cache. But doing a complete re-sync for such conditions is undesirable.
Even if we referesh the cache - there should probably be way to diff the sets - rather than re-create all the flows all over again.