[QoS] qos_plugin._extend_port_resource_request is killing port retrieval performance

Bug #1834484 reported by Rodolfo Alonso
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Rodolfo Alonso

Bug Description

The function [1], called during the port retrieval, is killing the performance. Although this code is needed for the BW aware scheduler (Nova-Neutron RFE), the high number of DB calls done inside this function is reducing the performance.

Instead of this, we propose:
- To publish inside the port DB object its own network qos policy, if any.
- Use it inside this function to retrieve only once, if needed, the QoS policy.

NOTE: the patches, if accepted, should be cherry-picked into Stein too.

NOTE: benchmarks: done with 60 ports in a development laptop. Command executed: "openstack port list"
- Stein and master: 6.5 secs
- Stein and master, deleting the code of this function: 1.5 secs
- Rocky: 1.5 secs.

[1] https://github.com/openstack/neutron/blob/stable/stein/neutron/services/qos/qos_plugin.py#L86

tags: added: qos
tags: added: stein-backport-potential
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
description: updated
Changed in neutron:
importance: Undecided → High
tags: added: api
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/667981

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/667998

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/667981
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=66fca96e5293040d23fa76380ba3f88b4117a5d7
Submitter: Zuul
Branch: master

commit 66fca96e5293040d23fa76380ba3f88b4117a5d7
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 15:02:33 2019 +0000

    Add qos_network_policy_id to Port OVO

    Added field "qos_network_policy_id" to Port OVO. This parameter
    will be used to retrieve the QoS policy bound to the port network.
    This reduces the number of calls to the database by creating a join
    between the QosNetworkPolicyBinding table and the Port table, based
    on the network ID.

    This backref association is not persistent (marked as "viewonly").
    This relationship is using for loading the QoS policy ID of the
    port network in the Port OVO.

    Related-Bug: #1834484

    Change-Id: I219a925d5e269b8c73a0481daa879d72c399fd8f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/stein)

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/669360

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/669394

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/667998
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8b00349a63c65e04b3d188366245234e880b7f60
Submitter: Zuul
Branch: master

commit 8b00349a63c65e04b3d188366245234e880b7f60
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 17:31:14 2019 +0000

    Refactor qos_plugin._extend_port_resource_request

    In order to speed up the port collection, some DB calls
    have been deleted in the extended function
    "QoSPlugin._extend_port_resource_request".

    With the parent commit, the port DB/OVO object now has the
    network QoS policy ID, making the network retrieval unneeded.

    In this refactor the QoS policy is collected from the DB only
    if a valid QoS policy ID exists, bound to the port or to the
    network. This reduces the number of QoS objects collection to
    one or zero.

    Change-Id: Iadf704c00378da99c502dca3e3b79796c368cac7
    Closes-Bug: #1834484

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/669360
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b452c508b6271e5809252aadd2332b60b41efe3b
Submitter: Zuul
Branch: stable/stein

commit b452c508b6271e5809252aadd2332b60b41efe3b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 15:02:33 2019 +0000

    Add qos_network_policy_id to Port OVO

    Added field "qos_network_policy_id" to Port OVO. This parameter
    will be used to retrieve the QoS policy bound to the port network.
    This reduces the number of calls to the database by creating a join
    between the QosNetworkPolicyBinding table and the Port table, based
    on the network ID.

    This backref association is not persistent (marked as "viewonly").
    This relationship is using for loading the QoS policy ID of the
    port network in the Port OVO.

    Related-Bug: #1834484

    Change-Id: I219a925d5e269b8c73a0481daa879d72c399fd8f
    (cherry picked from commit 66fca96e5293040d23fa76380ba3f88b4117a5d7)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/669394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1229be91d9c66ef073188fc1fa916d50e1710ba1
Submitter: Zuul
Branch: stable/stein

commit 1229be91d9c66ef073188fc1fa916d50e1710ba1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 17:31:14 2019 +0000

    Refactor qos_plugin._extend_port_resource_request

    In order to speed up the port collection, some DB calls
    have been deleted in the extended function
    "QoSPlugin._extend_port_resource_request".

    With the parent commit, the port DB/OVO object now has the
    network QoS policy ID, making the network retrieval unneeded.

    In this refactor the QoS policy is collected from the DB only
    if a valid QoS policy ID exists, bound to the port or to the
    network. This reduces the number of QoS objects collection to
    one or zero.

    Change-Id: Iadf704c00378da99c502dca3e3b79796c368cac7
    Closes-Bug: #1834484
    (cherry picked from commit 8b00349a63c65e04b3d188366245234e880b7f60)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/669908

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/669908
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a06e08de6c5ec18a32905b8e562526737ab2f18
Submitter: Zuul
Branch: master

commit 4a06e08de6c5ec18a32905b8e562526737ab2f18
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jul 9 16:51:19 2019 +0000

    Improve "QoSPlugin._validate_create_port_callback"

    Make use of the OVO Port new field "qos_network_policy_id" to avoid
    a DB call.

    Change-Id: I3724acfbe88ab3db1641c3edb5dd4c76bc84bc00
    Related-Bug: #1834484

tags: added: neutron-proactive-backport-potential
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Both in master and Stein. tag "neutron-proactive-backport-potential" can be removed now.

tags: removed: neutron-proactive-backport-potential stein-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.0.0.0b1

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 14.0.3

This issue was fixed in the openstack/neutron 14.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.