Configuration of the ovs controller by neutron-ovs-agent isn't idempotent
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
Medium
|
Slawek Kaplonski |
Bug Description
When neutron-ovs-agent is restarted, or doing full sync e.g. after recovery of the connectivity to rabbitmq, it is setting controller for bridges in the openvswitch. That operation isn't idempotent, and even if controller was already created, it will create new one. That, in some cases can lead to short (1-2 seconds) traffic loss in the dataplane. We observed that by doing UDP traffic test between 2 VMs:
- server A run:
sudo iperf3 -s -p 5000 -i 1
- server B run:
iperf3 -c 192.168.155.143 -p 5000 -u --length 1400 -b 1M -i 1 -t 3001
When neutron-ovs-agent on the node where server B runs is restarted, packet loss can be observed in iperf, like:
[ 5] 44.00-45.00 sec 122 KBytes 997 Kbits/sec 0.028 ms 0/89 (0%)
iperf3: OUT OF ORDER - incoming packet = 4047 and received packet = 4048 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 4046 and received packet = 4049 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 4051 and received packet = 4053 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 4052 and received packet = 4054 AND SP = 5
[ 5] 45.00-46.00 sec 123 KBytes 1.01 Mbits/sec 0.021 ms 4/90 (4.4%)
[ 5] 46.00-47.00 sec 122 KBytes 997 Kbits/sec 0.028 ms 0/89 (0%)
iperf3: OUT OF ORDER - incoming packet = 4218 and received packet = 4219 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 4220 and received packet = 4221 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 4222 and received packet = 4223 AND SP = 5
[ 5] 47.00-48.00 sec 122 KBytes 997 Kbits/sec 0.024 ms 3/89 (3.4%)
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master) | #1 |
Changed in neutron: | |
status: | Confirmed → In Progress |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master) | #2 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master) | #3 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 0194856da19f0ce
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master) | #4 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 11d166be683d36b
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
Changed in neutron: | |
status: | In Progress → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena) | #5 |
Fix proposed to branch: stable/xena
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby) | #6 |
Fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria) | #7 |
Fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri) | #8 |
Fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train) | #9 |
Fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein) | #10 |
Fix proposed to branch: stable/stein
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky) | #11 |
Fix proposed to branch: stable/rocky
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens) | #12 |
Fix proposed to branch: stable/queens
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena) | #13 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 0a945f626f38edb
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
tags: | added: in-stable-xena |
tags: | added: in-stable-wallaby |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby) | #14 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit b1eccf5a2deaf6d
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri) | #15 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit 11fe2bff17d8ad2
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
tags: | added: in-stable-ussuri |
tags: | added: in-stable-train |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train) | #16 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/train
commit 4456a87dac68d56
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
(cherry picked from commit 11fe2bff17d8ad2
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria) | #17 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit 18c1795e88ff89e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
tags: | added: in-stable-victoria |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein) | #18 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/stein
commit e2cc54977e27c30
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
(cherry picked from commit 11fe2bff17d8ad2
tags: | added: in-stable-stein |
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.4.2 | #19 |
This issue was fixed in the openstack/neutron 16.4.2 release.
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena) | #20 |
Related fix proposed to branch: stable/xena
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby) | #21 |
Related fix proposed to branch: stable/wallaby
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/victoria) | #22 |
Related fix proposed to branch: stable/victoria
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/victoria) | #23 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/victoria
commit f35a0a06e591ddb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Conflicts:
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
(cherry picked from commit 0194856da19f0ce
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena) | #24 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 2fcf48260ad85d4
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Conflicts:
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
(cherry picked from commit 0194856da19f0ce
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby) | #25 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/wallaby
commit cc0d42e0dfb94b5
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Conflicts:
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
(cherry picked from commit 0194856da19f0ce
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens) | #26 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/queens
commit d91dc006269b83f
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
(cherry picked from commit 11fe2bff17d8ad2
tags: | added: in-stable-queens |
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky) | #27 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/rocky
commit 0471901f7a9e315
Author: Slawek Kaplonski <email address hidden>
Date: Mon Oct 25 10:59:24 2021 +0200
Don't setup bridge controller if it is already set
Setting new controller for bridge every time when neutron-ovs-agent
is restarted or is doing full-sync may cause some short data plane
connectivity loss and is not needed if same controller is already
configured for the bridge.
With this patch neutron-ovs-agent will first check if controller is
configured for the bridge and if it's the same as what should be
configured, it will skip setting it up.
With this patch also protocols added to the bridge will be first checked
if they aren't already there and only missing ones will be added if
necessary.
Setting of the connectivity mode and inactivity probe is
always performed as this don't cause connectivity issues and is cheap
so we can always ensure that those parameters are configured properly.
Conflicts:
Closes-Bug: #1948642
Change-Id: Idfa763df8c60d8
(cherry picked from commit 11d166be683d36b
(cherry picked from commit b1eccf5a2deaf6d
(cherry picked from commit 11fe2bff17d8ad2
tags: | added: in-stable-rocky |
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.1.0 | #28 |
This issue was fixed in the openstack/neutron 19.1.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.3.0 | #29 |
This issue was fixed in the openstack/neutron 17.3.0 release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.2.0 | #30 |
This issue was fixed in the openstack/neutron 18.2.0 release.
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ussuri) | #31 |
Related fix proposed to branch: stable/ussuri
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/train) | #32 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ussuri) | #33 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/ussuri
commit 93d08bb575b879c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Conflicts:
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
(cherry picked from commit 0194856da19f0ce
(cherry picked from commit f35a0a06e591ddb
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/train) | #34 |
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/train
commit 426df27ad38c32e
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 26 09:21:33 2021 +0000
[OVS] Workaround when OpenFlow controller restarts
When a new OF controller is added or when new OF protocols are added to
a bridge, the OF controller is restarted. When that happens, the os-ken
``OfctlServ
for an update event that will add a new one. During this time no OF
controllers (datapaths) will be registered and any request will be
rejected with a ``InvalidDatapath`` exception.
This patch is a temporary workaround that prevents any command from
being executed just after those two described operations, providing
time to OVS to send the OF updates.
Conflicts:
Related-Bug: #1948642
Change-Id: I03878a2e650d5f
(cherry picked from commit 0194856da19f0ce
(cherry picked from commit f35a0a06e591ddb
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1 | #35 |
This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol | #36 |
This issue was fixed in the openstack/neutron queens-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol | #37 |
This issue was fixed in the openstack/neutron rocky-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron stein-eol | #38 |
This issue was fixed in the openstack/neutron stein-eol release.
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron train-eol | #39 |
This issue was fixed in the openstack/neutron train-eol release.
Fix proposed to branch: master /review. opendev. org/c/openstack /neutron/ +/815255
Review: https:/