neutron

[L2][scale issue] ovs-agent restart costs too long time

Bug #1813707 reported by LIU Yulong
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Unassigned

Bug Description

When subnets or security group ports quantity reach 2000+, the ovs-agent will take more than 15-40 mins+ to restart.
During this restart time, the ovs will not process any port, aka VM booting on this host will not get the L2 flows established.
This is a subproblem of bug #1813703, for more information, please see the summary:
https://bugs.launchpad.net/neutron/+bug/1813703

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

So do you know where the 15-4-mins is spent by the ovs-agent, is it just spending time in the polling loop or is it locked by some resource.

tags: added: ovs
Revision history for this message
LIU Yulong (dragon889) wrote :

Again this function setup_port_filters is mostly time-consuming:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1791-L1792
https://bugs.launchpad.net/neutron/+bug/1813704

But that retry fullsync can also waste time.
https://bugs.launchpad.net/neutron/+bug/1813706

Revision history for this message
Dirk Mueller (dmllr) wrote :

Is there a debug log available of that restart? it would be good to see the timings of

                LOG.debug("Update security group information for ports %s",
                          devices.keys())

and

                    LOG.debug("Update port filter for %s", device['device'])

in that setup.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/638641

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/638642

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/638645

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/638646

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/638642
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6ac420df7eb3ed324669472c61fec41b3d9cb35b
Submitter: Zuul
Branch: master

commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/649343

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/649365

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/649366

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/649369

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/638646
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8408af4f173a0ffde354599e26c49bf9e17e8bef
Submitter: Zuul
Branch: master

commit 8408af4f173a0ffde354599e26c49bf9e17e8bef
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/649682

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/649683

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/649688

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/649691

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649693

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/649701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/638645
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Submitter: Zuul
Branch: master

commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.openstack.org/650389

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/650390

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/650392

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/650393

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/650394

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein)

Reviewed: https://review.openstack.org/649343
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=98139553424375a2a0ec18fb6b07b4bf30fe88d0
Submitter: Zuul
Branch: stable/stein

commit 98139553424375a2a0ec18fb6b07b4bf30fe88d0
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)

tags: added: in-stable-stein
tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/649366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=195c1378317719d548dfd149ecc0ec9b01d53eef
Submitter: Zuul
Branch: stable/queens

commit 195c1378317719d548dfd149ecc0ec9b01d53eef
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/649369
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=51a766653395c11985b7dd5d3e3549224ae4ca88
Submitter: Zuul
Branch: stable/pike

commit 51a766653395c11985b7dd5d3e3549224ae4ca88
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Conflicts:
     neutron/agent/common/ovs_lib.py
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)
    (cherry picked from commit 5424b9a68cb3ac1fcc04ed8ae603c421bde2dee3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/649365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6494fcc2e44d9d9310e3ebaa92582f4f78d08b75
Submitter: Zuul
Branch: stable/rocky

commit 6494fcc2e44d9d9310e3ebaa92582f4f78d08b75
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein)

Reviewed: https://review.openstack.org/649682
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7d30ea950844f11348fa2827908622e3a8c7dfb
Submitter: Zuul
Branch: stable/stein

commit d7d30ea950844f11348fa2827908622e3a8c7dfb
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/650389
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7764064d0455634b18cc0931bcc44343913a1c6
Submitter: Zuul
Branch: stable/stein

commit d7764064d0455634b18cc0931bcc44343913a1c6
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/650390
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26a9765afb917901ca40e3117ff092774823ada2
Submitter: Zuul
Branch: stable/rocky

commit 26a9765afb917901ca40e3117ff092774823ada2
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/650392
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=df4e0a5394dff4cc176096abc64079d2c43fa9e7
Submitter: Zuul
Branch: stable/queens

commit df4e0a5394dff4cc176096abc64079d2c43fa9e7
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/650394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a4bc6e43fb7274b940fb88f13011821e283b3bb
Submitter: Zuul
Branch: stable/ocata

commit 7a4bc6e43fb7274b940fb88f13011821e283b3bb
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/650393
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Submitter: Zuul
Branch: stable/pike

commit bb508f9e6051ccb27b6dfda05b5c52b961f7370a
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:34:40 2019 +0800

    Change default local ovs connection timeout

    Large number of flows can cause local ovs connection
    timeout. Ultimately getting succeed will be better
    than a retry or fullsync.

    Related-Bug: #1813703
    Related-Bug: #1813705
    Related-Bug: #1813707
    Related-Bug: #1813709

    Change-Id: Ifa0608a7e131df3cad2f7727426720afce641a58
    (cherry picked from commit 64ea642359e8f8aee2ebe494e037ecdfe8cf1b2c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/649688
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=39afe0a129b6b979d0b56ec59048a4e16bedf7a9
Submitter: Zuul
Branch: stable/queens

commit 39afe0a129b6b979d0b56ec59048a4e16bedf7a9
Author: LIU Yulong <email address hidden>
Date: Fri Apr 12 18:47:24 2019 +0300

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
     neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)
    (cherry picked from commit 5d705468de1e495639f8b87266ccfc9391ce6135)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/649683
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d705468de1e495639f8b87266ccfc9391ce6135
Submitter: Zuul
Branch: stable/rocky

commit 5d705468de1e495639f8b87266ccfc9391ce6135
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.opendev.org/649691
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fa16540d2dd80f836c8fa2a424717899ac64af60
Submitter: Zuul
Branch: stable/pike

commit fa16540d2dd80f836c8fa2a424717899ac64af60
Author: LIU Yulong <email address hidden>
Date: Fri Apr 12 18:47:24 2019 +0300

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
     neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)
    (cherry picked from commit d7d30ea950844f11348fa2827908622e3a8c7dfb)
    (cherry picked from commit 5d705468de1e495639f8b87266ccfc9391ce6135)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.opendev.org/649701
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=49df07c7039206c537c17f40140d290e1b28a3f4
Submitter: Zuul
Branch: stable/ocata

commit 49df07c7039206c537c17f40140d290e1b28a3f4
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 16:47:42 2019 +0800

    Divide-and-conquer security group beasts

    In one specific compute node, the security group rules
    can be enormous quantity. This patch adds a step-by-step
    processing method to deal with the large number of the
    security group rules. And also changes or adds some LOG.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813707

    Conflicts:
     neutron/common/constants.py
            neutron/agent/common/ovs_lib.py
    Conflicts:
     neutron/agent/securitygroups_rpc.py
     neutron/common/constants.py
     neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Conflicts:
     neutron/agent/common/ovs_lib.py
     neutron/common/constants.py

    Change-Id: I57bf27ec75cf848271c5a28b22beee12b8bd5faa
    (cherry picked from commit 6ac420df7eb3ed324669472c61fec41b3d9cb35b)
    (cherry picked from commit f5d110e15f60753d056da942414ca6ecd6b78d8a)
    (cherry picked from commit 5424b9a68cb3ac1fcc04ed8ae603c421bde2dee3)

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/649693
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb2734b0d524aef348b69ae02988449f9dd63c56
Submitter: Zuul
Branch: stable/ocata

commit bb2734b0d524aef348b69ae02988449f9dd63c56
Author: LIU Yulong <email address hidden>
Date: Thu Feb 21 16:39:50 2019 +0800

    Do not call update_device_list in large sets

    Ovs-agent can process the ports in large sets, then all
    of these ports will have to update DB status or attributes.
    But neutron server is centralized. It may have to do
    something else, or the database processing can be also
    time-consuming. Because of these, it sometimes returns
    the RPC timeout exception to ovs-agent. And a fullsync
    will be triggered in next rpc loop. The restart time is
    becoming longer and longer.

    Adds a default step to update the port to reduce
    the probability of RPC timeout.

    Related-Bug: #1813703
    Related-Bug: #1813704
    Related-Bug: #1813706
    Related-Bug: #1813707

    Conflicts:
            neutron/common/constants.py
            neutron/agent/rpc.py
            neutron/tests/unit/plugins/ml2/test_rpc.py

    Change-Id: Ie37f4a4869969e235ce16b73cdfcbdc98626823e
    (cherry picked from commit 8408af4f173a0ffde354599e26c49bf9e17e8bef)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/638641
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8e73de8bc42067c0a6796df3cca9938d25ae754e
Submitter: Zuul
Branch: master

commit 8e73de8bc42067c0a6796df3cca9938d25ae754e
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/721239

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/721240

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/721242

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/721239
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a10413eb3fa52de2f330b165ac81c9dd47aeda57
Submitter: Zuul
Branch: stable/stein

commit a10413eb3fa52de2f330b165ac81c9dd47aeda57
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/721240
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=41fe9ff147244eb6d7573235fa12f45ed56be9b3
Submitter: Zuul
Branch: stable/rocky

commit 41fe9ff147244eb6d7573235fa12f45ed56be9b3
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/721242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=713ad71c6f4e389f3a224ce2898b09977b5045bb
Submitter: Zuul
Branch: stable/queens

commit 713ad71c6f4e389f3a224ce2898b09977b5045bb
Author: LIU Yulong <email address hidden>
Date: Wed Feb 20 14:01:08 2019 +0800

    Change ovs-agent iteration log level to INFO

    Operators may want to see how long it takes in the port
    processing procedure since DEBUG log does not enable
    basically in the production envrionment.

    Related-Bug: #1813703
    Related-Bug: #1813707
    Related-Bug: #1813706
    Related-Bug: #1813709

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I43733546abf5421d0e3f4cd5a959d279e1b89d1e
    (cherry picked from commit 8e73de8bc42067c0a6796df3cca9938d25ae754e)

Slawek Kaplonski (slaweq)
Changed in neutron:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/920819

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.