| 2016-12-12 15:52:23 |
Pierre Crégut |
bug |
|
|
added bug |
| 2016-12-12 15:52:23 |
Pierre Crégut |
attachment added |
|
Script reproducing the problem https://bugs.launchpad.net/bugs/1649317/+attachment/4790746/+files/combinatorial_blowup.sh |
|
| 2016-12-12 16:22:45 |
Armando Migliaccio |
neutron: status |
New |
Confirmed |
|
| 2016-12-12 16:22:52 |
Armando Migliaccio |
neutron: assignee |
|
Kevin Benton (kevinbenton) |
|
| 2016-12-12 16:22:55 |
Armando Migliaccio |
neutron: importance |
Undecided |
Low |
|
| 2016-12-12 16:31:43 |
Pierre Crégut |
description |
A regular tenant can create objects that will require a lot of time to enumerate because of the strategy used by the ORM to build back the object from the different tables in the database.
The script attached can be used to reproduce the problem. It creates a network with several subnetworks (with several outes and DNS servers), several tags, several RBAC policies but it does not exceed any typical quota. Because the network is retrieved at each stage it is modified, it is hard to run this script until its end in a typical settings.
Using the strategy lazy='joined' means that a single request is performed to retrieve an object and all its parts that may be expressed in several tables. For example when one asks for the network list, a complex query will be issued that also retrieves subnets, subnetpools, dns agents, etc. The exact query is visible at http://paste.openstack.org/show/592120/
Unfortunately using the strategy lazy=joined has another impact when the relation between the parent object and the sub-object has a ?-n arity. Rather than giving back exactly the row needed, the single query builds a kind of cross-product of the answers sharing the join keys. For example if we have a network with 4 tags and 4 subnetworks, we will have at least 16 rows for each combination of tags and subnetworks. Other fields like rbac rules, special routes, dns servers can amplify the problem.
It is not clear if the heavy usage of the database server and neutron server could lead to a real deny of service for other users. |
A regular tenant can create objects that will require a lot of time to enumerate because of the strategy used by the ORM to build back the objects from the different tables in the database.
The script attached can be used to reproduce the problem. It creates a network with several subnetworks (with several routes and DNS servers), several tags, several RBAC policies but it does not exceed any typical quota. Because the network is retrieved at each stage it is modified, it is hard to run this script until its end in a typical setup.
Using the strategy lazy='joined' means that a single request is performed to retrieve an object and all its parts that may be expressed in several tables. For example when one asks for the network list, a complex query will be issued that also retrieves subnets, subnetpools, dns agents, etc. The exact query is visible at http://paste.openstack.org/show/592120/
Unfortunately using the strategy lazy=joined has another impact when the relation between the parent object and the sub-object has a ?-n arity. Rather than giving back exactly the row needed, the single query builds a kind of cross-product of the answers sharing the join keys. For example if we have a network with 4 tags and 4 subnetworks, we will have at least 16 rows for each combination of tags and subnetworks. Other fields like rbac rules, special routes, dns servers can amplify the problem.
It is not clear if the heavy usage of the database server and neutron server could lead to a real deny of service for other users. |
|
| 2016-12-12 16:32:57 |
Pierre Crégut |
description |
A regular tenant can create objects that will require a lot of time to enumerate because of the strategy used by the ORM to build back the objects from the different tables in the database.
The script attached can be used to reproduce the problem. It creates a network with several subnetworks (with several routes and DNS servers), several tags, several RBAC policies but it does not exceed any typical quota. Because the network is retrieved at each stage it is modified, it is hard to run this script until its end in a typical setup.
Using the strategy lazy='joined' means that a single request is performed to retrieve an object and all its parts that may be expressed in several tables. For example when one asks for the network list, a complex query will be issued that also retrieves subnets, subnetpools, dns agents, etc. The exact query is visible at http://paste.openstack.org/show/592120/
Unfortunately using the strategy lazy=joined has another impact when the relation between the parent object and the sub-object has a ?-n arity. Rather than giving back exactly the row needed, the single query builds a kind of cross-product of the answers sharing the join keys. For example if we have a network with 4 tags and 4 subnetworks, we will have at least 16 rows for each combination of tags and subnetworks. Other fields like rbac rules, special routes, dns servers can amplify the problem.
It is not clear if the heavy usage of the database server and neutron server could lead to a real deny of service for other users. |
A regular tenant can create objects that will require a lot of time to enumerate because of the strategy used by the ORM to build back the objects from the different tables in the database.
The script attached can be used to reproduce the problem. It creates a network with several subnetworks (with several routes and DNS servers), several tags, several RBAC policies but it does not exceed any typical quota. Because the network is retrieved at each stage it is modified, it is hard to run this script until its end in a typical setup.
Using the strategy lazy='joined' means that a single request is performed to retrieve an object and all its parts that may be expressed in several tables. For example when one asks for the network list, a complex query will be issued that also retrieves subnets, subnetpools, dns agents, etc. The exact query is visible at http://paste.openstack.org/show/592120/
Unfortunately using the strategy lazy=joined has another impact when the relation between the parent object and the sub-object has a ?-n arity. Rather than giving back exactly the row needed, the single query builds a kind of cross-product of the answers sharing the join keys. For example if we have a network with 4 tags and 4 subnetworks, we will have at least 16 rows for each combination of tags and subnetworks. Other fields like rbac rules, special routes, dns servers can amplify the problem.
It is not clear if the heavy usage of the database server and neutron server could lead to a real denial of service for other users. |
|
| 2016-12-12 16:36:05 |
Thomas Morin |
bug |
|
|
added subscriber Kevin Benton |
| 2016-12-12 18:12:58 |
Kevin Benton |
neutron: importance |
Low |
High |
|
| 2016-12-12 19:47:26 |
OpenStack Infra |
neutron: status |
Confirmed |
In Progress |
|
| 2016-12-13 09:54:48 |
Thomas Morin |
bug |
|
|
added subscriber Thomas Morin |
| 2016-12-20 11:18:17 |
Piotr Misiak |
bug |
|
|
added subscriber Piotr Misiak |
| 2016-12-23 10:35:13 |
OpenStack Infra |
tags |
|
in-stable-newton |
|
| 2017-01-17 21:49:29 |
Ihar Hrachyshka |
tags |
in-stable-newton |
in-stable-newton neutron-proactive-backport-potential |
|
| 2017-02-02 00:19:15 |
Ihar Hrachyshka |
tags |
in-stable-newton neutron-proactive-backport-potential |
in-stable-newton neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
|
| 2017-02-02 00:24:57 |
Ihar Hrachyshka |
tags |
in-stable-newton neutron-easy-proactive-backport-potential neutron-proactive-backport-potential |
in-stable-newton |
|
| 2017-02-02 00:25:57 |
Ihar Hrachyshka |
tags |
in-stable-newton |
in-stable-newton neutron-proactive-backport-potential |
|
| 2017-02-03 15:26:05 |
Kevin Benton |
tags |
in-stable-newton neutron-proactive-backport-potential |
in-stable-newton neutron-proactive-backport-potential ocata-rc-potential |
|
| 2017-02-05 16:13:08 |
Armando Migliaccio |
neutron: milestone |
|
ocata-rc1 |
|
| 2017-02-07 07:10:09 |
OpenStack Infra |
neutron: status |
In Progress |
Fix Released |
|
| 2017-02-16 23:17:54 |
OpenStack Infra |
tags |
in-stable-newton neutron-proactive-backport-potential ocata-rc-potential |
in-stable-newton in-stable-ocata neutron-proactive-backport-potential ocata-rc-potential |
|
| 2017-03-31 08:24:04 |
Daniel Alvarez |
tags |
in-stable-newton in-stable-ocata neutron-proactive-backport-potential ocata-rc-potential |
in-stable-newton in-stable-ocata ocata-rc-potential |
|
