[L2][scale issue] ovs-agent restart costs too long time

[LIU Yulong]

When subnets or security group ports quantity reach 2000+, the ovs-agent will take more than 15-40 mins+ to restart.
During this restart time, the ovs will not process any port, aka VM booting on this host will not get the L2 flows established.
This is a subproblem of bug #1813703, for more information, please see the summary:
https://bugs.launchpad.net/neutron/+bug/1813703

[Swaminathan Vasudevan (swaminathan-vasudevan)]

So do you know where the 15-4-mins is spent by the ovs-agent, is it just spending time in the polling loop or is it locked by some resource.

[LIU Yulong (dragon889)]

Again this function setup_port_filters is mostly time-consuming:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1791-L1792
https://bugs.launchpad.net/neutron/+bug/1813704

But that retry fullsync can also waste time.
https://bugs.launchpad.net/neutron/+bug/1813706

[Dirk Mueller (dmllr)]

Is there a debug log available of that restart? it would be good to see the timings of

                LOG.debug("Update security group information for ports %s",
                          devices.keys())

and

                    LOG.debug("Update port filter for %s", device['device'])

in that setup.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638641

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638642

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638645

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/638646

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638642
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6ac420df7eb3ed324669472c61fec41b3d9cb35b
Submitter: Zuul
Branch: master

commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/649343

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/649365

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/649366

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/649369

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638646
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8408af4f173a0ffde354599e26c49bf9e17e8bef
Submitter: Zuul
Branch: master

commit 8408af4f173a0ffde354599e26c49bf9e17e8bef
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/649682

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/649683

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/649688

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/649691

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649693

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638645
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Submitter: Zuul
Branch: master

commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/650389

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/650390

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/650392

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/650393

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/650394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649343
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=98139553424375a2a0ec18fb6b07b4bf30fe88d0
Submitter: Zuul
Branch: stable/stein

commit 98139553424375a2a0ec18fb6b07b4bf30fe88d0
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=195c1378317719d548dfd149ecc0ec9b01d53eef
Submitter: Zuul
Branch: stable/queens

commit 195c1378317719d548dfd149ecc0ec9b01d53eef
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649369
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=51a766653395c11985b7dd5d3e3549224ae4ca88
Submitter: Zuul
Branch: stable/pike

commit 51a766653395c11985b7dd5d3e3549224ae4ca88
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Conflicts:
     neutron/agent/common/ovs_lib.py
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)
    (cherry picked from commit 5424b9a68cb3ac1fcc04ed8ae603c421bde2dee3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6494fcc2e44d9d9310e3ebaa92582f4f78d08b75
Submitter: Zuul
Branch: stable/rocky

commit 6494fcc2e44d9d9310e3ebaa92582f4f78d08b75
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649682
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7d30ea950844f11348fa2827908622e3a8c7dfb
Submitter: Zuul
Branch: stable/stein

commit d7d30ea950844f11348fa2827908622e3a8c7dfb
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7764064d0455634b18cc0931bcc44343913a1c6
Submitter: Zuul
Branch: stable/stein

commit d7764064d0455634b18cc0931bcc44343913a1c6
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650390
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26a9765afb917901ca40e3117ff092774823ada2
Submitter: Zuul
Branch: stable/rocky

commit 26a9765afb917901ca40e3117ff092774823ada2
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650392
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=df4e0a5394dff4cc176096abc64079d2c43fa9e7
Submitter: Zuul
Branch: stable/queens

commit df4e0a5394dff4cc176096abc64079d2c43fa9e7
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a4bc6e43fb7274b940fb88f13011821e283b3bb
Submitter: Zuul
Branch: stable/ocata

commit 7a4bc6e43fb7274b940fb88f13011821e283b3bb
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/650393
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Submitter: Zuul
Branch: stable/pike

commit bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649688
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=39afe0a129b6b979d0b56ec59048a4e16bedf7a9
Submitter: Zuul
Branch: stable/queens

commit 39afe0a129b6b979d0b56ec59048a4e16bedf7a9
Author: LIU Yulong <email address hidden>
Date: Fri Apr 12 18:47:24 2019 +0300

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
     neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)
    (cherry picked from commit 5d705468de1e495639f8b87266ccfc9391ce6135)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/649683
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d705468de1e495639f8b87266ccfc9391ce6135
Submitter: Zuul
Branch: stable/rocky

commit 5d705468de1e495639f8b87266ccfc9391ce6135
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/649691
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fa16540d2dd80f836c8fa2a424717899ac64af60
Submitter: Zuul
Branch: stable/pike

commit fa16540d2dd80f836c8fa2a424717899ac64af60
Author: LIU Yulong <email address hidden>
Date: Fri Apr 12 18:47:24 2019 +0300

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
     neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)
    (cherry picked from commit 5d705468de1e495639f8b87266ccfc9391ce6135)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/649701
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=49df07c7039206c537c17f40140d290e1b28a3f4
Submitter: Zuul
Branch: stable/ocata

commit 49df07c7039206c537c17f40140d290e1b28a3f4
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py
            neutron/agent/common/ovs_lib.py
    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Conflicts:
     neutron/agent/common/ovs_lib.py
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)
    (cherry picked from commit 5424b9a68cb3ac1fcc04ed8ae603c421bde2dee3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/649693
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb2734b0d524aef348b69ae02988449f9dd63c56
Submitter: Zuul
Branch: stable/ocata

commit bb2734b0d524aef348b69ae02988449f9dd63c56
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
            neutron/common/constants.py
            neutron/agent/rpc.py
            neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/638641
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e73de8bc42067c0a6796df3cca9938d25ae754e
Submitter: Zuul
Branch: master

commit 8e73de8bc42067c0a6796df3cca9938d25ae754e
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/721239

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/721240

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/721242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721239
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a10413eb3fa52de2f330b165ac81c9dd47aeda57
Submitter: Zuul
Branch: stable/stein

commit a10413eb3fa52de2f330b165ac81c9dd47aeda57
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721240
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=41fe9ff147244eb6d7573235fa12f45ed56be9b3
Submitter: Zuul
Branch: stable/rocky

commit 41fe9ff147244eb6d7573235fa12f45ed56be9b3
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=713ad71c6f4e389f3a224ce2898b09977b5045bb
Submitter: Zuul
Branch: stable/queens

commit 713ad71c6f4e389f3a224ce2898b09977b5045bb
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920819

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920819
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.