Security Group extension reads all Neutron ports for anything other that a single server

[Phil Day]

Although https://review.openstack.org/#/c/30048/ optimized the
SecurityGroupsOutputController for the case where the server
list only contains one server, but in all other cases the current
code calls the Neutron driver in a way that makes it retrieve
all ports and security groups visible to the user.

For users with a Neutron admin role this retrieves all ports
and SecGroups in the system, which on a large system is a
major performance issue and often leads to client timeouts.
Normally these users have further qualified their query to a
specific tenant or host, or maybe just trying to get their own
list of servers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47651

[Joe Gordon (jogo)]

Adding tempest to this because, we should update

https://github.com/openstack/tempest/blob/master/tempest/scenario/test_large_ops.py

which we gate on to test this scenario.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47651
Committed: http://github.com/openstack/nova/commit/18c3ac4a8935b9997dbb55181d5dbb5232ef2c27
Submitter: Jenkins
Branch: master

commit 18c3ac4a8935b9997dbb55181d5dbb5232ef2c27
Author: Phil Day <email address hidden>
Date: Fri Sep 20 22:41:14 2013 +0000

    Fix performance of Server List with Neutron for Admins

    https://review.openstack.org/#/c/30048/ optimized the
    SecurityGroupsOutputController for the case where the server
    list only contains one server, but in all other cases the current
    code calls the Neutron driver in a way that makes it retrieve
    all ports and security groups visible to the tenant.

    For users with a Neutron admin role this retrieves all ports
    and all SecGroups in the system, which on a large system is a
    major performance issue and often leads to client timeouts.
    Normally these users have further qualified their query to a
    specific tenant or host, or are maybe just trying to get their
    own list of servers.

    If instead we pass the pre-filtered list of servers to Neutron
    it can query for ports associated with just the device IDs of
    the servers. We then use that list of ports to build a
    specific list of security groups IDs to query for.

    This approach means that we can remove the initial
    optimization for a single server case.

    Closes-Bug: 1228384

    Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/61924

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/61924
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=19fdaa225abd007a13cd38c742e27c5ee620186c
Submitter: Jenkins
Branch: stable/havana

commit 19fdaa225abd007a13cd38c742e27c5ee620186c
Author: Phil Day <email address hidden>
Date: Fri Sep 20 22:41:14 2013 +0000

    Fix performance of Server List with Neutron for Admins

    https://review.openstack.org/#/c/30048/ optimized the
    SecurityGroupsOutputController for the case where the server
    list only contains one server, but in all other cases the current
    code calls the Neutron driver in a way that makes it retrieve
    all ports and security groups visible to the tenant.

    For users with a Neutron admin role this retrieves all ports
    and all SecGroups in the system, which on a large system is a
    major performance issue and often leads to client timeouts.
    Normally these users have further qualified their query to a
    specific tenant or host, or are maybe just trying to get their
    own list of servers.

    If instead we pass the pre-filtered list of servers to Neutron
    it can query for ports associated with just the device IDs of
    the servers. We then use that list of ports to build a
    specific list of security groups IDs to query for.

    This approach means that we can remove the initial
    optimization for a single server case.

    Closes-Bug: 1228384

    (cherry picked from commit 18c3ac4a8935b9997dbb55181d5dbb5232ef2c27)

    Conflicts:
     nova/tests/api/openstack/compute/contrib/test_neutron_security_groups.py

    Change-Id: I6f72533056e4f336f7578cc883fd1a125c2048a9

[Luz Cazares (luz-cazares)]

Invalidating Tempest bug since scenario "tempest/scenario/test_large_ops.py" was removed from tempest (related to performance instead of scenario). See: https://github.com/openstack/tempest/commit/1976da83a5bdb35a61d1659bb5ece1b5d248bacd