[L2][scale issue] ovs-agent dump-flows takes a lots of time

[LIU Yulong]

ovs-agent clean stale flows action will dump all the bridge flows first. When subnets or security group ports quantity reach 2000+, this will become really time-consuming.
And sometimes this dump action can also get failed, then the ovs-agent will dump again. And things get worse.
This is a subproblem of bug #1813703, for more information, please see the summary:
https://bugs.launchpad.net/neutron/+bug/1813703

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I think you are talking about this
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py#L170
So you are suggesting that we should be cleaning the stale flows without a lookup, is it possible to clean without a lookup.

[LIU Yulong (dragon889)]

For now, we have no solution for this issue, but IMO we shoud divide-and-conquer the entire flows. It is really a giant beast.

[LIU Yulong (dragon889)]

dump-flows failed with timeout:

2019-02-20 14:55:23.957 10229 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-63c61d58-2a38-47fa-ba4a-9b74fd38d9fa - - - - -] Cleaning stale br-int flows
2019-02-20 14:57:25.450 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-63c61d58-2a38-47fa-ba4a-9b74fd38d9fa - - - - -] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x548ae654,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) timed out: Timeout: 120 seconds
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-63c61d58-2a38-47fa-ba4a-9b74fd38d9fa - - - - -] Error while processing VIF ports: RuntimeError: ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x548ae654,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=255,type=1) timed out
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2120, in rpc_loop
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.cleanup_stale_flows()
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1872, in cleanup_stale_flows
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent bridge.cleanup_flows()
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 154, in cleanup_flows
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent cookies = set([f.cookie for f in self.dump_flows()]) - \
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 147, in dump_flows
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent reply_multi=True)
2019-02-20 14:57:25.451 10229 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/of...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638645

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638647

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638647
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f898ffd71fba4f9b8fd9f4cb851fc3976d72396a
Submitter: Zuul
Branch: master

commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/648207

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/648217

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/648219

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/648220

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649414

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638645
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Submitter: Zuul
Branch: master

commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/650389

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/650390

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/650392

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/650393

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/650394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648219
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e4bfc7d50ee94502bead86078a123676bc9c24f9
Submitter: Zuul
Branch: stable/queens

commit e4bfc7d50ee94502bead86078a123676bc9c24f9
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb
    (cherry picked from commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648220
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fb84771d1364d9be6fa7d0bce1bc89b2e3541271
Submitter: Zuul
Branch: stable/pike

commit fb84771d1364d9be6fa7d0bce1bc89b2e3541271
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb
    (cherry picked from commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648217
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=af67d516a5b39b883fa6fb2fca4673fb7602b292
Submitter: Zuul
Branch: stable/rocky

commit af67d516a5b39b883fa6fb2fca4673fb7602b292
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb
    (cherry picked from commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/648207
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7865264aaba615f6e52f5806d844531696186d56
Submitter: Zuul
Branch: stable/stein

commit 7865264aaba615f6e52f5806d844531696186d56
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb
    (cherry picked from commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649414
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ea3d844c75541cc2be17865bad6336cd1b8385c4
Submitter: Zuul
Branch: stable/ocata

commit ea3d844c75541cc2be17865bad6336cd1b8385c4
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 19:46:53 2019 +0800

    Divide-and-conquer local bridge flows beasts

    The dump-flows action will get a very large sets of flow information
    if there are enormous ports or openflow security group rules. For now
    we can meet some known exception during such action, for instance,
    memory issue, timeout issue.
    So after this patch, the cleanup action of the bridge stale flows
    will be done one table by one table. But note, this only supports
    for 'native' OpenFlow interface driver.

    Related-Bug: #1813703
    Related-Bug: #1813712
    Related-Bug: #1813709
    Related-Bug: #1813708

    Change-Id: Ie06d1bebe83ffeaf7130dcbb8ca21e5e59a220fb
    (cherry picked from commit f898ffd71fba4f9b8fd9f4cb851fc3976d72396a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7764064d0455634b18cc0931bcc44343913a1c6
Submitter: Zuul
Branch: stable/stein

commit d7764064d0455634b18cc0931bcc44343913a1c6
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650390
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26a9765afb917901ca40e3117ff092774823ada2
Submitter: Zuul
Branch: stable/rocky

commit 26a9765afb917901ca40e3117ff092774823ada2
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650392
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=df4e0a5394dff4cc176096abc64079d2c43fa9e7
Submitter: Zuul
Branch: stable/queens

commit df4e0a5394dff4cc176096abc64079d2c43fa9e7
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a4bc6e43fb7274b940fb88f13011821e283b3bb
Submitter: Zuul
Branch: stable/ocata

commit 7a4bc6e43fb7274b940fb88f13011821e283b3bb
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650393
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Submitter: Zuul
Branch: stable/pike

commit bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/638641
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e73de8bc42067c0a6796df3cca9938d25ae754e
Submitter: Zuul
Branch: master

commit 8e73de8bc42067c0a6796df3cca9938d25ae754e
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/721239

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/721240

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/721242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721239
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a10413eb3fa52de2f330b165ac81c9dd47aeda57
Submitter: Zuul
Branch: stable/stein

commit a10413eb3fa52de2f330b165ac81c9dd47aeda57
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721240
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=41fe9ff147244eb6d7573235fa12f45ed56be9b3
Submitter: Zuul
Branch: stable/rocky

commit 41fe9ff147244eb6d7573235fa12f45ed56be9b3
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=713ad71c6f4e389f3a224ce2898b09977b5045bb
Submitter: Zuul
Branch: stable/queens

commit 713ad71c6f4e389f3a224ce2898b09977b5045bb
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920819

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920819
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.