get_networks performance hindered by segment lookups

[Kevin Benton]

During the get_networks method of ML2, we iterate over each network and do a database call to lookup the segments for that network. This scales the number of database calls linearly with the number of retrieved networks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/256846

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/257222

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/256846
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=806e67538fbea49530aa0eedb75694cd8bad439d
Submitter: Jenkins
Branch: master

commit 806e67538fbea49530aa0eedb75694cd8bad439d
Author: Kevin Benton <email address hidden>
Date: Fri Dec 11 10:55:38 2015 -0800

    Batch db segment retrieval

    A net-list operation was calling extend_network_dict_provider for
    each network individually which would result in a database call for
    each network.

    This adds a new call in the manager to extend multiple networks at
    once and then it adds a bulk version of get_network_segments that
    it calls.

    Now 1 net list of any number of networks will only result in 1
    segment DB call.

    Change-Id: I2543b3bdbb178ee4bb8d1288e9a27af1c5c8c8b4
    Closes-Bug: #1525423
    Partial-Bug: #1513782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/258343

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/258343
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7c1fb08045f86e36b4971acdc61f9712d56f0345
Submitter: Jenkins
Branch: stable/liberty

commit 7c1fb08045f86e36b4971acdc61f9712d56f0345
Author: Kevin Benton <email address hidden>
Date: Fri Dec 11 10:55:38 2015 -0800

    Batch db segment retrieval

    A net-list operation was calling extend_network_dict_provider for
    each network individually which would result in a database call for
    each network.

    This adds a new call in the manager to extend multiple networks at
    once and then it adds a bulk version of get_network_segments that
    it calls.

    Now 1 net list of any number of networks will only result in 1
    segment DB call.

    Change-Id: I2543b3bdbb178ee4bb8d1288e9a27af1c5c8c8b4
    Closes-Bug: #1525423
    Partial-Bug: #1513782
    (cherry picked from commit 806e67538fbea49530aa0eedb75694cd8bad439d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/257222
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4885ef4885de072b0321858a2ae334b708da4bea
Submitter: Jenkins
Branch: master

commit 4885ef4885de072b0321858a2ae334b708da4bea
Author: Kevin Benton <email address hidden>
Date: Mon Dec 14 00:44:16 2015 -0800

    Add tests that constrain db query count

    This patch adds unit tests to ML2 and L3 that ensure that the
    number of DB calls during list operations for ports, networks,
    subnets, routers, and floating IPs remains constant regardless
    of the number of ports.

    These will prevent changes from slipping in that result in
    a separate DB query for each object in a list operation
    (for changes to the extensions used by ML2 and the DVR plugin).

    Related-Bug: #1525295
    Related-Bug: #1513782
    Related-Bug: #1525423
    Related-Bug: #1525740
    Related-Bug: #1526644

    Change-Id: I1958fc7c318bbf73238a3ad5be133fa7800c8290

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/267467

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/267467
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0d5d7c72feaa0a25b799632cefcd873933d2a28b
Submitter: Jenkins
Branch: stable/liberty

commit 0d5d7c72feaa0a25b799632cefcd873933d2a28b
Author: Kevin Benton <email address hidden>
Date: Mon Dec 14 00:44:16 2015 -0800

    Add tests that constrain db query count

    This patch adds unit tests to ML2 and L3 that ensure that the
    number of DB calls during list operations for ports, networks,
    subnets, routers, and floating IPs remains constant regardless
    of the number of ports.

    These will prevent changes from slipping in that result in
    a separate DB query for each object in a list operation
    (for changes to the extensions used by ML2 and the DVR plugin).

    Related-Bug: #1525295
    Related-Bug: #1513782
    Related-Bug: #1525423
    Related-Bug: #1525740
    Related-Bug: #1526644

    Conflicts:
     doc/source/devref/effective_neutron.rst
     neutron/tests/unit/extensions/test_l3.py

    Change-Id: I1958fc7c318bbf73238a3ad5be133fa7800c8290
    (cherry picked from commit 4885ef4885de072b0321858a2ae334b708da4bea)