SG list retrieval is slow because we are always loading "synthetic_fields"

Bug #2052419 reported by Rodolfo Alonso
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Committed
Medium
Rodolfo Alonso

Bug Description

Since [1], we introduced a way to skip the load of the OVO synthetic fields depending on the resource fields retrieved. In the case of the security groups (SG), the SG rules are child objects to the SGs. The SG rules are retrieved when a SG OVO is created.

The improvement done in [1] is to make the SG rules load dynamically, that means using the load mode "lazy='dynamic'". That will issue a SQL query only if the SG rules are read; if not, the query is never issued.

However since [3] (and this is previous to the [1] optimization), we always add the field "shared" to the filters and thus to the fields to retrieve, because it is a policy required field. Because "shared" is a synthetic field [4], that will force the SG "synthetic_fields" load and the retrieval of the SG rules always. This is undoing any performance improvement.

Because the "shared" object belongs to the RBAC functionality and we are always going to load it, we need another way to load the "shared" synthetic field *without* loading the others (if no other synthetic field is required, as is for example the "os security group list" command).

Bugzilla reference (OSP16.2): https://bugzilla.redhat.com/show_bug.cgi?id=2262100

[1]https://review.opendev.org/q/topic:%22bug/1810563%22
[2]https://github.com/openstack/neutron/blob/b85b19e3846fa74975ba5d703336b6e7cd8af433/neutron/db/models/securitygroup.py#L90-L93
[3]https://review.opendev.org/c/openstack/neutron/+/328313
[4]https://github.com/openstack/neutron/blob/b85b19e3846fa74975ba5d703336b6e7cd8af433/neutron/objects/rbac_db.py#L349

Changed in neutron:
importance: Undecided → Medium
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/908571

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/2023.2)

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/908586

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/2023.1)

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/908587

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/908588

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :
Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/908571
Committed: https://opendev.org/openstack/neutron/commit/52662cad7a07f9bb3b954366eeccd9795ae21503
Submitter: "Zuul (22348)"
Branch: master

commit 52662cad7a07f9bb3b954366eeccd9795ae21503
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Feb 8 21:56:06 2024 +0000

    Change SG rules backref load method to "joined"

    Since [1], we introduced a way to skip the load of the OVO synthetic
    fields depending on the resource fields retrieved. In the case of the
    security groups (SG), the SG rules are child objects to the SGs. The SG
    rules are retrieved when a SG OVO is created.

    The improvement done in [1] is to make the SG rules load dynamically,
    that means using the load mode "lazy='dynamic'". That will issue a SQL
    query only if the SG rules are read; if not, the query is never issued.

    However since [2] (and this is previous to the [1] optimization), we
    always add the field "shared" to the filters and thus to the fields to
    retrieve, because it is a policy required field. Because "shared" is a
    synthetic field [3], that will force the SG "synthetic_fields" load and
    the retrieval of the SG rules always. This is undoing any performance
    improvement.

    This patch is changing the loading method to "joined"; that will request
    the SG rules in the same SQL query, instead of issuing separate queries
    for the SG rules. Until a method to load the SG "shared" field,
    independently of the synthetic OVO fields is implemented, this change
    will improve the SG retrieval performance. In a testing environment
    with around 5500K SG and 4 rules (default ones) per SG:
    * lazy='dynamic': 38 seconds
    * lazy='select': 20 seconds
    * lazy='joined': 12 seconds

    [1]https://review.opendev.org/q/topic:%22bug/1810563%22
    [2]https://review.opendev.org/c/openstack/neutron/+/328313
    [3]https://github.com/openstack/neutron/blob/b85b19e3846fa74975ba5d703336b6e7cd8af433/neutron/objects/rbac_db.py#L349

    Related-Bug: #2052419
    Change-Id: I300464472f2348d148f2a3ddac384c883d9d815b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/908586
Committed: https://opendev.org/openstack/neutron/commit/aad8dd867744cee76ec90b314bea70989ca8cae9
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit aad8dd867744cee76ec90b314bea70989ca8cae9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Feb 8 21:56:06 2024 +0000

    Change SG rules backref load method to "joined"

    Since [1], we introduced a way to skip the load of the OVO synthetic
    fields depending on the resource fields retrieved. In the case of the
    security groups (SG), the SG rules are child objects to the SGs. The SG
    rules are retrieved when a SG OVO is created.

    The improvement done in [1] is to make the SG rules load dynamically,
    that means using the load mode "lazy='dynamic'". That will issue a SQL
    query only if the SG rules are read; if not, the query is never issued.

    However since [2] (and this is previous to the [1] optimization), we
    always add the field "shared" to the filters and thus to the fields to
    retrieve, because it is a policy required field. Because "shared" is a
    synthetic field [3], that will force the SG "synthetic_fields" load and
    the retrieval of the SG rules always. This is undoing any performance
    improvement.

    This patch is changing the loading method to "joined"; that will request
    the SG rules in the same SQL query, instead of issuing separate queries
    for the SG rules. Until a method to load the SG "shared" field,
    independently of the synthetic OVO fields is implemented, this change
    will improve the SG retrieval performance. In a testing environment
    with around 5500K SG and 4 rules (default ones) per SG:
    * lazy='dynamic': 38 seconds
    * lazy='select': 20 seconds
    * lazy='joined': 12 seconds

    [1]https://review.opendev.org/q/topic:%22bug/1810563%22
    [2]https://review.opendev.org/c/openstack/neutron/+/328313
    [3]https://github.com/openstack/neutron/blob/b85b19e3846fa74975ba5d703336b6e7cd8af433/neutron/objects/rbac_db.py#L349

    Related-Bug: #2052419
    Change-Id: I300464472f2348d148f2a3ddac384c883d9d815b
    (cherry picked from commit 52662cad7a07f9bb3b954366eeccd9795ae21503)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/908587
Committed: https://opendev.org/openstack/neutron/commit/3dec91ec58c0c9e986b8ed704936410ef5999c38
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 3dec91ec58c0c9e986b8ed704936410ef5999c38
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Feb 8 21:56:06 2024 +0000

    Change SG rules backref load method to "joined"

    Since [1], we introduced a way to skip the load of the OVO synthetic
    fields depending on the resource fields retrieved. In the case of the
    security groups (SG), the SG rules are child objects to the SGs. The SG
    rules are retrieved when a SG OVO is created.

    The improvement done in [1] is to make the SG rules load dynamically,
    that means using the load mode "lazy='dynamic'". That will issue a SQL
    query only if the SG rules are read; if not, the query is never issued.

    However since [2] (and this is previous to the [1] optimization), we
    always add the field "shared" to the filters and thus to the fields to
    retrieve, because it is a policy required field. Because "shared" is a
    synthetic field [3], that will force the SG "synthetic_fields" load and
    the retrieval of the SG rules always. This is undoing any performance
    improvement.

    This patch is changing the loading method to "joined"; that will request
    the SG rules in the same SQL query, instead of issuing separate queries
    for the SG rules. Until a method to load the SG "shared" field,
    independently of the synthetic OVO fields is implemented, this change
    will improve the SG retrieval performance. In a testing environment
    with around 5500K SG and 4 rules (default ones) per SG:
    * lazy='dynamic': 38 seconds
    * lazy='select': 20 seconds
    * lazy='joined': 12 seconds

    [1]https://review.opendev.org/q/topic:%22bug/1810563%22
    [2]https://review.opendev.org/c/openstack/neutron/+/328313
    [3]https://github.com/openstack/neutron/blob/b85b19e3846fa74975ba5d703336b6e7cd8af433/neutron/objects/rbac_db.py#L349

    Related-Bug: #2052419
    Change-Id: I300464472f2348d148f2a3ddac384c883d9d815b
    (cherry picked from commit 52662cad7a07f9bb3b954366eeccd9795ae21503)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/zed)

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/908588

Changed in neutron:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.