| 2022-05-25 07:21:18 |
Henning Eggers |
bug |
|
|
added bug |
| 2022-05-25 09:18:47 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2022-05-25 15:48:44 |
Bernard Cafarelli |
tags |
|
ovs-fw |
|
| 2022-05-25 15:48:48 |
Bernard Cafarelli |
neutron: importance |
Undecided |
Medium |
|
| 2022-05-31 07:54:01 |
Lajos Katona |
neutron: assignee |
|
Henning Eggers (henninge) |
|
| 2022-06-08 09:31:38 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2022-06-10 10:32:14 |
OpenStack Infra |
tags |
ovs-fw |
in-stable-yoga ovs-fw |
|
| 2022-06-10 10:46:26 |
OpenStack Infra |
tags |
in-stable-yoga ovs-fw |
in-stable-xena in-stable-yoga ovs-fw |
|
| 2022-06-10 13:10:04 |
OpenStack Infra |
tags |
in-stable-xena in-stable-yoga ovs-fw |
in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
|
| 2022-06-10 13:10:10 |
OpenStack Infra |
tags |
in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
in-stable-ussuri in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
|
| 2022-06-10 13:10:16 |
OpenStack Infra |
tags |
in-stable-ussuri in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
|
| 2022-06-10 19:24:15 |
OpenStack Infra |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-xena in-stable-yoga ovs-fw |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw |
|
| 2022-09-08 04:05:42 |
Hua Zhang |
attachment added |
|
focal.debdiff https://bugs.launchpad.net/neutron/+bug/1975674/+attachment/5614519/+files/focal.debdiff |
|
| 2022-09-08 04:06:35 |
Hua Zhang |
summary |
Neutron agent blocks during VM deletion when a remote security group is involved |
[SRU] Neutron agent blocks during VM deletion when a remote security group is involved |
|
| 2022-09-08 04:08:05 |
Hua Zhang |
description |
When deleting a VM that has a security group referring to a remote security group, the neutron agent will block for as long as it takes to remove the respective flows. This happens when the remote security group contains many (thousands) ports referring to other VMs.
Steps to reproduce:
- Create a VM with security group A
- Add a rule to security group A allowing access from a remote security group B
- Add a large number or ports to security group B (e.g. 2000)
- The respective ovs flows will be added
- Delete the VM
- The ovs flows will be removed
Expected:
- VM and flow to be deleted within seconds
- No impact to other VMs on the same hypervisor
Actual:
- Flow deletion takes a long time, sometimes up to 10 minutes
- While flows are being deleted, no VMs can be created on the same hypervisor
The reason for this behavior is that under the hood the agent calls ovs-ofctl (via execve()) once for each port in the remote security group. These calls quickly add up to minutes if there are many ports.
The proposed solution would be to use deferred execution for the flow deletion. In that case it becomes a bulk operation and around 400 flows are deleted in one call. In addition it runs in the background and does not block the agent for other operations. |
When deleting a VM that has a security group referring to a remote security group, the neutron agent will block for as long as it takes to remove the respective flows. This happens when the remote security group contains many (thousands) ports referring to other VMs.
Steps to reproduce:
- Create a VM with security group A
- Add a rule to security group A allowing access from a remote security group B
- Add a large number or ports to security group B (e.g. 2000)
- The respective ovs flows will be added
- Delete the VM
- The ovs flows will be removed
Expected:
- VM and flow to be deleted within seconds
- No impact to other VMs on the same hypervisor
Actual:
- Flow deletion takes a long time, sometimes up to 10 minutes
- While flows are being deleted, no VMs can be created on the same hypervisor
The reason for this behavior is that under the hood the agent calls ovs-ofctl (via execve()) once for each port in the remote security group. These calls quickly add up to minutes if there are many ports.
The proposed solution would be to use deferred execution for the flow deletion. In that case it becomes a bulk operation and around 400 flows are deleted in one call. In addition it runs in the background and does not block the agent for other operations.
[Impact]
Please see LP bug description for full details.
[Test Plan]
Please see the section 'Steps to reproduce in LP bug description.
[Regression Potential]
The fix[1] is already in the upstream stable/ussuri, here's just SRU into 16.4.2, so it's a clean backport.
[1] https://opendev.org/openstack/neutron/commit/30ef996f8aa0b0bc57a280690871f1081946ffee |
|
| 2022-09-08 04:08:40 |
Hua Zhang |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed |
|
| 2022-09-15 12:44:32 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu) |
|
| 2022-09-15 12:55:27 |
Corey Bryant |
nominated for series |
|
Ubuntu Focal |
|
| 2022-09-15 12:55:27 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Focal) |
|
| 2022-09-15 12:55:45 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2022-09-15 12:56:00 |
Corey Bryant |
nominated for series |
|
cloud-archive/ussuri |
|
| 2022-09-15 12:56:00 |
Corey Bryant |
bug task added |
|
cloud-archive/ussuri |
|
| 2022-09-15 12:56:00 |
Corey Bryant |
nominated for series |
|
cloud-archive/victoria |
|
| 2022-09-15 12:56:00 |
Corey Bryant |
bug task added |
|
cloud-archive/victoria |
|
| 2022-09-15 12:56:11 |
Corey Bryant |
cloud-archive: status |
New |
Invalid |
|
| 2022-09-15 12:56:17 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2022-09-15 12:56:21 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Triaged |
|
| 2022-09-15 12:57:15 |
Corey Bryant |
neutron (Ubuntu Focal): status |
New |
Triaged |
|
| 2022-09-15 12:57:19 |
Corey Bryant |
neutron (Ubuntu): status |
New |
Invalid |
|
| 2022-09-15 12:58:14 |
Corey Bryant |
neutron (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2022-09-15 12:58:17 |
Corey Bryant |
cloud-archive/ussuri: importance |
Undecided |
Medium |
|
| 2022-09-15 12:58:21 |
Corey Bryant |
cloud-archive/victoria: importance |
Undecided |
Medium |
|
| 2022-09-15 13:00:49 |
Corey Bryant |
description |
When deleting a VM that has a security group referring to a remote security group, the neutron agent will block for as long as it takes to remove the respective flows. This happens when the remote security group contains many (thousands) ports referring to other VMs.
Steps to reproduce:
- Create a VM with security group A
- Add a rule to security group A allowing access from a remote security group B
- Add a large number or ports to security group B (e.g. 2000)
- The respective ovs flows will be added
- Delete the VM
- The ovs flows will be removed
Expected:
- VM and flow to be deleted within seconds
- No impact to other VMs on the same hypervisor
Actual:
- Flow deletion takes a long time, sometimes up to 10 minutes
- While flows are being deleted, no VMs can be created on the same hypervisor
The reason for this behavior is that under the hood the agent calls ovs-ofctl (via execve()) once for each port in the remote security group. These calls quickly add up to minutes if there are many ports.
The proposed solution would be to use deferred execution for the flow deletion. In that case it becomes a bulk operation and around 400 flows are deleted in one call. In addition it runs in the background and does not block the agent for other operations.
[Impact]
Please see LP bug description for full details.
[Test Plan]
Please see the section 'Steps to reproduce in LP bug description.
[Regression Potential]
The fix[1] is already in the upstream stable/ussuri, here's just SRU into 16.4.2, so it's a clean backport.
[1] https://opendev.org/openstack/neutron/commit/30ef996f8aa0b0bc57a280690871f1081946ffee |
When deleting a VM that has a security group referring to a remote security group, the neutron agent will block for as long as it takes to remove the respective flows. This happens when the remote security group contains many (thousands) ports referring to other VMs.
Steps to reproduce:
- Create a VM with security group A
- Add a rule to security group A allowing access from a remote security group B
- Add a large number or ports to security group B (e.g. 2000)
- The respective ovs flows will be added
- Delete the VM
- The ovs flows will be removed
Expected:
- VM and flow to be deleted within seconds
- No impact to other VMs on the same hypervisor
Actual:
- Flow deletion takes a long time, sometimes up to 10 minutes
- While flows are being deleted, no VMs can be created on the same hypervisor
The reason for this behavior is that under the hood the agent calls ovs-ofctl (via execve()) once for each port in the remote security group. These calls quickly add up to minutes if there are many ports.
The proposed solution would be to use deferred execution for the flow deletion. In that case it becomes a bulk operation and around 400 flows are deleted in one call. In addition it runs in the background and does not block the agent for other operations.
[Impact]
Please see LP bug description for full details.
[Test Plan]
Please see the section 'Steps to reproduce in LP bug description.
[Regression Potential]
This is fixed in ubuntu jammy and in cloud archive wallaby+ releases. The SRU will include fixes for usuri/victoria cloud archives and ubuntu focal. The fix[1] is already in the upstream stable branches.
[1] https://opendev.org/openstack/neutron/commit/30ef996f8aa0b0bc57a280690871f1081946ffee |
|
| 2022-09-16 12:24:21 |
Corey Bryant |
cloud-archive/victoria: status |
Triaged |
Fix Committed |
|
| 2022-09-16 12:24:23 |
Corey Bryant |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-victoria-needed |
|
| 2022-09-16 12:24:58 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-09-23 16:15:15 |
Brian Murray |
neutron (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2022-09-23 16:15:22 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2022-09-23 16:15:31 |
Brian Murray |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-victoria-needed |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-victoria-needed |
|
| 2022-09-23 20:48:16 |
Corey Bryant |
cloud-archive/ussuri: status |
Triaged |
Fix Committed |
|
| 2022-09-23 20:48:19 |
Corey Bryant |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-victoria-needed |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed |
|
| 2022-09-27 10:06:08 |
Hua Zhang |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-done |
|
| 2022-09-28 01:07:54 |
Hua Zhang |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-done |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-done-focal verification-needed verification-ussuri-needed verification-victoria-done |
|
| 2022-09-28 04:46:16 |
Hua Zhang |
tags |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-done-focal verification-needed verification-ussuri-needed verification-victoria-done |
in-stable-train in-stable-ussuri in-stable-victoria in-stable-wallaby in-stable-xena in-stable-yoga ovs-fw sts sts-sru-needed verification-done verification-done-focal verification-ussuri-done verification-victoria-done |
|
| 2022-10-03 18:54:07 |
Corey Bryant |
cloud-archive/victoria: status |
Fix Committed |
Fix Released |
|
| 2022-10-05 03:28:07 |
Launchpad Janitor |
neutron (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2022-10-05 12:00:57 |
Corey Bryant |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
