Listing ports creates inefficient database query

Bug #2016704 reported by Ihtisham ul Haq
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Ihtisham ul Haq

Bug Description

Listing ports in a project with large number of ports results in querying the database which isn't efficient

Steps to reproduce:
1. Create 2000 ports in 20 different networks(the more the better) in the same project.
2. Enable query log for mysql using `slow_query_log` and setting `long_query_time` to 0.1 seconds
3. Now list the ports using `openstack port list` as non-admin user
4. The following query should be logged in the `slow_query_log_file`: https://paste.opendev.org/show/819649/
5. Here is part of the query that is relevant for the bug:
```
SELECT DISTINCT ports.project_id AS ports_project_id, ports.id AS ports_id, ports.name AS ports_name, ports.network_id AS ports_network_id, ports.mac_address AS ports_mac_address, ports.admin_state_up AS ports_admin_state_up, ports.status AS ports_status, ports.device_id AS ports_device_id, ports.device_owner AS ports_device_owner, ports.ip_allocation AS ports_ip_allocation, ports.standard_attr_id AS ports_standard_attr_id
FROM ports, networks
WHERE ports.project_id = '<project-id>' OR ports.network_id = networks.id AND networks.project_id = '<project-id>' ORDER BY ports.id ASC
```

6. Executing the above query responds in about 0.25 seconds <-- Which is the problem.

7. Now update this query to use *explicit join* instead of implicit on:
```
SELECT DISTINCT ports.project_id AS ports_project_id, ports.id AS ports_id, ports.name AS ports_name, ports.network_id
AS ports_network_id, ports.mac_address AS ports_mac_address, ports.admin_state_up AS ports_admin_state_up, ports.status AS ports_status, ports.device_id AS ports_device_id, ports.device_owner AS ports_device_owner, ports.ip_allocation AS ports_ip_allocation, ports.standard_attr_id AS ports_standard_attr_id
FROM ports JOIN networks ON ports.network_id = networks.id
WHERE ports.project_id = '<project-id>' AND networks.project_id = '<project-id>' ORDER BY ports.id ASC;
```

8. The response time of this is around 0.02 seconds

On our production environment(Yoga version) we see an extreme scenario of this, we have one project which has about 350 ports and 2 networks. And for that project, the query responds in 15 seconds, where as the optimized one responds in 0.01 seconds.

In total we have about 1000 projects and most of the projects have about 10 ports.

Version:
- Openstack version, neutron from trunk. CommitID 28961c8b76a4b09412825231a3f69374b183aefd
- Single node devstack on Ubuntu 22.04.02 LTS

Ihtisham ul Haq (iulhaq)
description: updated
description: updated
Ihtisham ul Haq (iulhaq)
description: updated
Changed in neutron:
status: New → In Progress
Ihtisham ul Haq (iulhaq)
description: updated
Ihtisham ul Haq (iulhaq)
description: updated
Revision history for this message
Ihtisham ul Haq (iulhaq) wrote :
Changed in neutron:
importance: Undecided → High
Changed in neutron:
assignee: nobody → Ihtisham ul Haq (iulhaq)
Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I believe I have reported the same behavior here https://bugs.launchpad.net/neutron/+bug/1973349
And can't say that patch provided helped really a lot, but slightly improved situation.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/880621
Committed: https://opendev.org/openstack/neutron/commit/f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c
Submitter: "Zuul (22348)"
Branch: master

commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/883712

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/883713

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/883714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/883912

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/883913

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883714
Committed: https://opendev.org/openstack/neutron/commit/edbb0f4aa5896ecdd4c5356b592a09827af3b3b8
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit edbb0f4aa5896ecdd4c5356b592a09827af3b3b8
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6
    (cherry picked from commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883712
Committed: https://opendev.org/openstack/neutron/commit/508d8328aa52b3f4bc6dff9ab1c4715631221c41
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 508d8328aa52b3f4bc6dff9ab1c4715631221c41
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6
    (cherry picked from commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883912
Committed: https://opendev.org/openstack/neutron/commit/fcfe550d5b164b6521eae0de514fc961bba35bb0
Submitter: "Zuul (22348)"
Branch: stable/xena

commit fcfe550d5b164b6521eae0de514fc961bba35bb0
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6
    (cherry picked from commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883713
Committed: https://opendev.org/openstack/neutron/commit/004ed33ec1ab549444e518ed8681b51737b81b9c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 004ed33ec1ab549444e518ed8681b51737b81b9c
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6
    (cherry picked from commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883913
Committed: https://opendev.org/openstack/neutron/commit/dfe6d8b4b4cd1c3b14f152b6f7c603ed0c485175
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit dfe6d8b4b4cd1c3b14f152b6f7c603ed0c485175
Author: Ihtisham ul Haq <email address hidden>
Date: Mon Apr 17 12:23:30 2023 +0200

    Use explicit inner join for networks in port query

    This improves the performance of the database when fetching a list of ports
    for a project user. This change creates an inner join with the networks
    belonging to the ports.

    Previous SQL query:
    SELECT ports ...
    FROM network, ports ...
    WHERE ports.project_id = <project>
    OR ports.network_id = networks.id
    AND networks.project_id = <project>

    Current SQL query:
    SELECT ports ...
    FROM ports
    INNER JOIN networks ON networks.id = ports.network_id
    WHERE ports.project_id = <project>
    OR networks.project_id = <project>

    Closes-Bug: #2016704
    Change-Id: I9c49a307956ecfbf8bd2e866cefb21a212c38bd6
    (cherry picked from commit f23d7af8d7e36b9d52a8c8ccf64355e22f3adb9c)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 22.0.1

This issue was fixed in the openstack/neutron 22.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.1.1

This issue was fixed in the openstack/neutron 21.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.3.1

This issue was fixed in the openstack/neutron 20.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 23.0.0.0b3

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron wallaby-eom

This issue was fixed in the openstack/neutron wallaby-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron xena-eom

This issue was fixed in the openstack/neutron xena-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.