Slow metadata API performance with security groups that have a lot of rules

Bug #1851430 reported by Matt Riedemann on 2019-11-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Doug Wiegley
Pike
Low
Lee Yarwood
Queens
Low
Matt Riedemann
Rocky
Medium
Matt Riedemann
Stein
Medium
Matt Riedemann
Train
Medium
Matt Riedemann

Bug Description

This was reported here without a bug:

https://review.opendev.org/#/c/656084/

The EC2 metadata API response includes a 'security-groups' key that is a list of security group names attached to the instance.

The problem is for each security group attached to the instance, if the group has a lot of rules on it, it can be expensive to query (join) that information from neutron, especially if we don't care about the rules.

By default, listing security groups includes the rules in the response:

https://docs.openstack.org/api-ref/network/v2/index.html?expanded=list-security-groups-detail#list-security-groups

For the purpose of the EC2 metadata API, we should just query security groups for their names.

Matt Riedemann (mriedem) on 2019-11-05
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Matt Riedemann (mriedem) on 2019-11-05
Changed in nova:
assignee: nobody → Doug Wiegley (dougwig)
Changed in nova:
assignee: Doug Wiegley (dougwig) → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Matt Riedemann (mriedem) on 2019-11-05
Changed in nova:
assignee: Matt Riedemann (mriedem) → Doug Wiegley (dougwig)
summary: - slow metadata performance with security groups that have a lot of rules
+ Slow metadata API performance with security groups that have a lot of
+ rules

Reviewed: https://review.opendev.org/656084
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eaf16fdde59a14fb38df669b21a911a0c2d2576f
Submitter: Zuul
Branch: master

commit eaf16fdde59a14fb38df669b21a911a0c2d2576f
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/694409
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=418af2d865809cfa907678f883dae07f4f31baa2
Submitter: Zuul
Branch: stable/train

commit 418af2d865809cfa907678f883dae07f4f31baa2
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)

Reviewed: https://review.opendev.org/695925
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fec95a2e4f763e15193504483383f918feb3e636
Submitter: Zuul
Branch: stable/stein

commit fec95a2e4f763e15193504483383f918feb3e636
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    NOTE(mriedem): The test had to be modified in this backport because
    the mock call() behavior is different since Train and I'm not
    entirely sure why but I'm guessing it's due to change
    I4484e63c97bd1cdde3d88855eabe7545784f365e which was added in Train.
    To get around this, I just modified the test to use a simple
    predictable side_effect for _convert_to_nova_security_group_format.

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)

Reviewed: https://review.opendev.org/697517
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=38b2f68a17533e839819e654825613aefd4effd4
Submitter: Zuul
Branch: stable/rocky

commit 38b2f68a17533e839819e654825613aefd4effd4
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)

Reviewed: https://review.opendev.org/697518
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=00d438adb325610a04af9f8f18cdb1c622df5418
Submitter: Zuul
Branch: stable/queens

commit 00d438adb325610a04af9f8f18cdb1c622df5418
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)
    (cherry picked from commit 38b2f68a17533e839819e654825613aefd4effd4)

This issue was fixed in the openstack/nova 20.1.0 release.

This issue was fixed in the openstack/nova 19.1.0 release.

This issue was fixed in the openstack/nova 18.3.0 release.

Reviewed: https://review.opendev.org/697523
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=13969dc32a4dedfe4ee0fbebec4b5491cf9f984b
Submitter: Zuul
Branch: stable/pike

commit 13969dc32a4dedfe4ee0fbebec4b5491cf9f984b
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Conflicts:
          nova/tests/unit/network/security_group/test_neutron_driver.py

    NOTE(mriedem): The conflict is due to not having change
    I31c9ea8628c6f3985f8e9118d9687bbfb8789b68 in Pike.

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)
    (cherry picked from commit 38b2f68a17533e839819e654825613aefd4effd4)
    (cherry picked from commit 00d438adb325610a04af9f8f18cdb1c622df5418)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers