Comment 6 for bug 1817022

Did some more scale testing and found increasing inactivity_probe can stop InvalidDatapath errors.

Using OVS firewall driver.
Add 100 rules to default security group.
Add 100 ports to host, one at a time slowly.
This results in about 24.5K flows in br-int. No problems to far.

Stop ovs-agent. Wait a couple of min.
Start ovs-agent.
Get InvalidDatapath errors and sync never completes. CPU very high.

Stop ovs-agent.
Set new of_inactivity_probe=10
Start ovs-agent
No errors and syncs after a couple of min. CPU drops to very low.

Stop ovs-agent
Set debug=False and of_inactivity_probe back to default 5.
Start ovs-agent. No problems.

Rebuilt OVS to allow inactivity_probe < 5 sec.
Set of_inactivity_probe=1 and debug=False
Start ovs-agent
Get InvalidDatapath errors and sync never completes. CPU very high.

2019-03-11T16:35:25.258Z|25948|rconn|ERR|br-int<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting
2019-03-11T16:35:25.258Z|25949|rconn|ERR|br-ex<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting
2019-03-11T16:35:25.258Z|25950|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting

Mar 11 16:35:23 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Cleaning stale br-int flows
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Cleaning stale br-ex flows
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR OfctlService [-] unknown dpid 104209607453507
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507: InvalidDatapath: Datapath Invalid 104209607453507
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Error while processing VIF ports: RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2230, in rpc_loop
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.cleanup_stale_flows()
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 160, in wrapper
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1974, in cleanup_stale_flows
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent bridge.cleanup_flows()
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 170, in cleanup_flows
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent cookies = set([f.cookie for f in self.dump_flows()]) - \
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 163, in dump_flows
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent reply_multi=True)
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 100, in _send_msg
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(m)
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507
Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
Mar 11 16:35:26 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Agent out of sync with plugin!