Did some more scale testing and found increasing inactivity_probe can stop InvalidDatapath errors. Using OVS firewall driver. Add 100 rules to default security group. Add 100 ports to host, one at a time slowly. This results in about 24.5K flows in br-int. No problems to far. Stop ovs-agent. Wait a couple of min. Start ovs-agent. Get InvalidDatapath errors and sync never completes. CPU very high. Stop ovs-agent. Set new of_inactivity_probe=10 Start ovs-agent No errors and syncs after a couple of min. CPU drops to very low. Stop ovs-agent Set debug=False and of_inactivity_probe back to default 5. Start ovs-agent. No problems. Rebuilt OVS to allow inactivity_probe < 5 sec. Set of_inactivity_probe=1 and debug=False Start ovs-agent Get InvalidDatapath errors and sync never completes. CPU very high. 2019-03-11T16:35:25.258Z|25948|rconn|ERR|br-int<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting 2019-03-11T16:35:25.258Z|25949|rconn|ERR|br-ex<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting 2019-03-11T16:35:25.258Z|25950|rconn|ERR|br-tun<->tcp:127.0.0.1:6633: no response to inactivity probe after 1 seconds, disconnecting Mar 11 16:35:23 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Cleaning stale br-int flows Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Cleaning stale br-ex flows Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR OfctlService [-] unknown dpid 104209607453507 Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507: InvalidDatapath: Datapath Invalid 104209607453507 Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Error while processing VIF ports: RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507 Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2230, in rpc_loop Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.cleanup_stale_flows() Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 160, in wrapper Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs) Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1974, in cleanup_stale_flows Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent bridge.cleanup_flows() Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 170, in cleanup_flows Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent cookies = set([f.cookie for f in self.dump_flows()]) - \ Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 163, in dump_flows Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent reply_multi=True) Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 100, in _send_msg Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(m) Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: ofctl request version=None,msg_type=None,msg_len=None,xid=None,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) error Datapath Invalid 104209607453507 Mar 11 16:35:25 ubuntu neutron-openvswitch-agent[9883]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Mar 11 16:35:26 ubuntu neutron-openvswitch-agent[9883]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-aadec15d-63f8-4f9f-b135-ec34c2d852a4 None None] Agent out of sync with plugin!