[QoS] qos_plugin._extend_port_resource_request is killing port retrieval performance

[Rodolfo Alonso]

The function [1], called during the port retrieval, is killing the performance. Although this code is needed for the BW aware scheduler (Nova-Neutron RFE), the high number of DB calls done inside this function is reducing the performance.

Instead of this, we propose:
- To publish inside the port DB object its own network qos policy, if any.
- Use it inside this function to retrieve only once, if needed, the QoS policy.

NOTE: the patches, if accepted, should be cherry-picked into Stein too.

NOTE: benchmarks: done with 60 ports in a development laptop. Command executed: "openstack port list"
- Stein and master: 6.5 secs
- Stein and master, deleting the code of this function: 1.5 secs
- Rocky: 1.5 secs.

[1] https://github.com/openstack/neutron/blob/stable/stein/neutron/services/qos/qos_plugin.py#L86

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/667981

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/667998

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667981
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=66fca96e5293040d23fa76380ba3f88b4117a5d7
Submitter: Zuul
Branch: master

commit 66fca96e5293040d23fa76380ba3f88b4117a5d7
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 15:02:33 2019 +0000

    Add qos_network_policy_id to Port OVO

    Added field "qos_network_policy_id" to Port OVO. This parameter
    will be used to retrieve the QoS policy bound to the port network.
    This reduces the number of calls to the database by creating a join
    between the QosNetworkPolicyBinding table and the Port table, based
    on the network ID.

    This backref association is not persistent (marked as "viewonly").
    This relationship is using for loading the QoS policy ID of the
    port network in the Port OVO.

    Related-Bug: #1834484

    Change-Id: I219a925d5e269b8c73a0481daa879d72c399fd8f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/669360

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/669394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667998
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8b00349a63c65e04b3d188366245234e880b7f60
Submitter: Zuul
Branch: master

commit 8b00349a63c65e04b3d188366245234e880b7f60
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 17:31:14 2019 +0000

    Refactor qos_plugin._extend_port_resource_request

    In order to speed up the port collection, some DB calls
    have been deleted in the extended function
    "QoSPlugin._extend_port_resource_request".

    With the parent commit, the port DB/OVO object now has the
    network QoS policy ID, making the network retrieval unneeded.

    In this refactor the QoS policy is collected from the DB only
    if a valid QoS policy ID exists, bound to the port or to the
    network. This reduces the number of QoS objects collection to
    one or zero.

    Change-Id: Iadf704c00378da99c502dca3e3b79796c368cac7
    Closes-Bug: #1834484

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669360
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b452c508b6271e5809252aadd2332b60b41efe3b
Submitter: Zuul
Branch: stable/stein

commit b452c508b6271e5809252aadd2332b60b41efe3b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 15:02:33 2019 +0000

    Add qos_network_policy_id to Port OVO

    Added field "qos_network_policy_id" to Port OVO. This parameter
    will be used to retrieve the QoS policy bound to the port network.
    This reduces the number of calls to the database by creating a join
    between the QosNetworkPolicyBinding table and the Port table, based
    on the network ID.

    This backref association is not persistent (marked as "viewonly").
    This relationship is using for loading the QoS policy ID of the
    port network in the Port OVO.

    Related-Bug: #1834484

    Change-Id: I219a925d5e269b8c73a0481daa879d72c399fd8f
    (cherry picked from commit 66fca96e5293040d23fa76380ba3f88b4117a5d7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1229be91d9c66ef073188fc1fa916d50e1710ba1
Submitter: Zuul
Branch: stable/stein

commit 1229be91d9c66ef073188fc1fa916d50e1710ba1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 27 17:31:14 2019 +0000

    Refactor qos_plugin._extend_port_resource_request

    In order to speed up the port collection, some DB calls
    have been deleted in the extended function
    "QoSPlugin._extend_port_resource_request".

    With the parent commit, the port DB/OVO object now has the
    network QoS policy ID, making the network retrieval unneeded.

    In this refactor the QoS policy is collected from the DB only
    if a valid QoS policy ID exists, bound to the port or to the
    network. This reduces the number of QoS objects collection to
    one or zero.

    Change-Id: Iadf704c00378da99c502dca3e3b79796c368cac7
    Closes-Bug: #1834484
    (cherry picked from commit 8b00349a63c65e04b3d188366245234e880b7f60)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/669908

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669908
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a06e08de6c5ec18a32905b8e562526737ab2f18
Submitter: Zuul
Branch: master

commit 4a06e08de6c5ec18a32905b8e562526737ab2f18
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jul 9 16:51:19 2019 +0000

    Improve "QoSPlugin._validate_create_port_callback"

    Make use of the OVO Port new field "qos_network_policy_id" to avoid
    a DB call.

    Change-Id: I3724acfbe88ab3db1641c3edb5dd4c76bc84bc00
    Related-Bug: #1834484

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Both in master and Stein. tag "neutron-proactive-backport-potential" can be removed now.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.3 release.