| 2021-03-08 14:47:00 |
LEDUC Florian |
bug |
|
|
added bug |
| 2021-03-08 14:55:42 |
LEDUC Florian |
description |
* Summary: Slownesses on neutron API with many RBAC rules
* High level description: Sharing several networks or security groups to project drastically increase API response time on some routes (/networks or /server/detail).
For quite some time we have observing that reponse times are increasing (slowly fur surely) on /networks calls. We have increased the number of Neutron workers, but in vain.
Lately, we're observing that it's getting worse (reponse time form 5 to 370 seconds). We discarded possible bottlenecks one by one (our service endpoint performance, neutron API configuration, etc).
But we have found that some calls in the DB takes a lot of time. It seems they are stuck in the mariadb database (10.3.10). So we have captured a slow queries in mysql.
An example of for /server/detail:
---------------------------------
http://paste.openstack.org/show/803334/
We can see that there are more than 2 millions of rows examinated, and around 1657 returned.
An example of for /networks:
----------------------------
http://paste.openstack.org/show/803337/
Rows_sent: 517 Rows_examined: 223519
* Pre-conditions:
Database tables size:
table:
- networkrbacs 16928 rows
- securitygrouprbacs 1691 rows
- keystone.project 1713 rows
Control plane nodes are shared with some others services:
- RMQ
- mariadb
- Openstack APIs
- DHCP agents
It seems the code of those lines are based on https://github.com/openstack/neutron-lib/blob/698e4c8daa7d43018a71122ec5b0cd5b17b55141/neutron_lib/db/model_query.py#L120
* Step-by-step reproduction steps:
- Create a lot of projects (at least 1000)
- Create a SG in admin account
- Create fake networks (vlan, vxlan) with associated
- Share the SG and all networks with all projects
* Expected output: lower response time, less than 5 seconds (approximatively).
* Actual output: May lead to gateway timeout.
* Version:
** OpenStack version Stein releases for all components
** CentOS 7.4 with kolla containers
** kolla-ansible for stein release
* Environment: We operate all services in Openstack except for Cinder.
* Perceived severity: Medium |
* Summary: Slownesses on neutron API with many RBAC rules
* High level description: Sharing several networks or security groups to project drastically increase API response time on some routes (/networks or /server/detail).
For quite some time we have observing that reponse times are increasing (slowly fur surely) on /networks calls. We have increased the number of Neutron workers, but in vain.
Lately, we're observing that it's getting worse (reponse time form 5 to 370 seconds). We discarded possible bottlenecks one by one (our service endpoint performance, neutron API configuration, etc).
But we have found that some calls in the DB takes a lot of time. It seems they are stuck in the mariadb database (10.3.10). So we have captured a slow queries in mysql.
An example of for /server/detail:
---------------------------------
http://paste.openstack.org/show/803334/
We can see that there are more than 2 millions of rows examinated, and around 1657 returned.
An example of for /networks:
----------------------------
http://paste.openstack.org/show/803337/
Rows_sent: 517 Rows_examined: 223519
* Pre-conditions:
Database tables size:
table:
- networkrbacs 16928 rows
- securitygrouprbacs 1691 rows
- keystone.project 1713 rows
Control plane nodes are shared with some others services:
- RMQ
- mariadb
- Openstack APIs
- DHCP agents
It seems the code of those lines are based on https://github.com/openstack/neutron-lib/blob/698e4c8daa7d43018a71122ec5b0cd5b17b55141/neutron_lib/db/model_query.py#L120
* Step-by-step reproduction steps:
- Create a lot of projects (at least 1000)
- Create a SG in admin account
- Create fake networks (vlan, vxlan) with associated
- Share the SG and all networks with all projects
* Expected output: lower response time, less than 5 seconds (approximatively).
* Actual output: May lead to gateway timeout.
* Version:
** OpenStack version Stein releases for all components (neutron 14.2.0).
** CentOS 7.4 with kolla containers
** kolla-ansible for stein release
* Environment: We operate all services in Openstack except for Cinder.
* Perceived severity: Medium |
|
| 2021-03-13 04:51:48 |
Hongbin Lu |
neutron: status |
New |
Confirmed |
|
| 2021-03-13 04:51:52 |
Hongbin Lu |
neutron: importance |
Undecided |
Medium |
|
| 2021-03-13 04:52:52 |
Hongbin Lu |
tags |
|
db |
|
| 2021-03-16 08:33:32 |
Slawek Kaplonski |
tags |
db |
db loadimpact |
|
| 2021-04-05 16:14:45 |
Rodolfo Alonso |
bug |
|
|
added subscriber Rodolfo Alonso |
| 2021-04-06 06:38:48 |
Slawek Kaplonski |
neutron: status |
Confirmed |
Incomplete |
|
| 2021-04-29 11:22:05 |
LEDUC Florian |
neutron: status |
Incomplete |
Confirmed |
|
| 2021-05-07 16:15:07 |
Tyler Stachecki |
bug |
|
|
added subscriber Tyler Stachecki |
| 2021-05-07 16:17:57 |
Tyler Stachecki |
neutron: assignee |
|
Tyler Stachecki (tstachecki) |
|
| 2021-06-01 07:13:36 |
LEDUC Florian |
attachment added |
|
db_model_query.diff.txt https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5501581/+files/db_model_query.diff.txt |
|
| 2021-06-02 03:37:58 |
Tyler Stachecki |
attachment added |
|
preliminary-neutron-rbac-perf.patch https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5501805/+files/preliminary-neutron-rbac-perf.patch |
|
| 2021-09-20 13:00:19 |
LEDUC Florian |
attachment added |
|
PERFORMANCE_ISSUE_20092021.txt https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5526474/+files/PERFORMANCE_ISSUE_20092021.txt |
|
| 2021-09-20 16:06:35 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2021-09-21 14:47:30 |
LEDUC Florian |
attachment removed |
PERFORMANCE_ISSUE_20092021.txt https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5526474/+files/PERFORMANCE_ISSUE_20092021.txt |
|
|
| 2021-09-21 14:49:39 |
LEDUC Florian |
attachment added |
|
Hostname removed from the last attachment. https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5526691/+files/PERFORMANCE_ISSUE_20092021.txt |
|
| 2021-10-02 10:57:12 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2021-10-28 20:04:07 |
Tyler Stachecki |
attachment removed |
preliminary-neutron-rbac-perf.patch https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5501805/+files/preliminary-neutron-rbac-perf.patch |
|
|
| 2021-10-28 20:05:42 |
Tyler Stachecki |
attachment added |
|
ussuri-v1.patch https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5536840/+files/ussuri-v1.patch |
|
| 2021-10-29 13:32:06 |
Tyler Stachecki |
attachment added |
|
ussuri-v2.patch https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5537010/+files/ussuri-v2.patch |
|
| 2021-10-29 13:32:22 |
Tyler Stachecki |
attachment removed |
ussuri-v1.patch https://bugs.launchpad.net/neutron/+bug/1918145/+attachment/5536840/+files/ussuri-v1.patch |
|
|
| 2022-04-14 19:35:02 |
Slawek Kaplonski |
neutron: assignee |
Tyler Stachecki (tstachecki) |
|
|
| 2022-04-14 19:35:07 |
Slawek Kaplonski |
tags |
db loadimpact |
db loadimpact timeout-abandon |
|
| 2022-06-13 03:06:04 |
Tyler Stachecki |
neutron: assignee |
|
Tyler Stachecki (tstachecki) |
|
| 2022-06-21 09:05:31 |
Clement G |
bug |
|
|
added subscriber Clement G |
| 2023-01-06 13:55:28 |
Sahid Orentino |
neutron: status |
Fix Released |
Confirmed |
|
| 2023-01-16 09:03:45 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2023-05-31 11:24:21 |
Rodolfo Alonso |
neutron: assignee |
Tyler Stachecki (tstachecki) |
Rodolfo Alonso (rodolfo-alonso-hernandez) |
|
| 2023-06-02 12:07:26 |
OpenStack Infra |
tags |
db loadimpact timeout-abandon |
db in-stable-zed loadimpact timeout-abandon |
|
| 2023-06-02 12:13:25 |
OpenStack Infra |
tags |
db in-stable-zed loadimpact timeout-abandon |
db in-stable-wallaby in-stable-zed loadimpact timeout-abandon |
|
| 2023-06-02 12:13:30 |
OpenStack Infra |
tags |
db in-stable-wallaby in-stable-zed loadimpact timeout-abandon |
db in-stable-wallaby in-stable-yoga in-stable-zed loadimpact timeout-abandon |
|
| 2023-06-02 12:13:35 |
OpenStack Infra |
tags |
db in-stable-wallaby in-stable-yoga in-stable-zed loadimpact timeout-abandon |
db in-stable-wallaby in-stable-xena in-stable-yoga in-stable-zed loadimpact timeout-abandon |
|
| 2023-06-04 23:31:14 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2023-11-09 10:59:22 |
Christian Rohmann |
bug |
|
|
added subscriber Christian Rohmann |
