Hyper-V Neutron Agent takes too long to initialise after restart

[Claudiu Belu]

On hosts with many switch ports, the Hyper-V Neutron Agent takes too long to initialise. This is due to the high number of switch ports, as the agent verifies each if it's bound to the vSwitch, if it has VLAN bound, has metrics enabled, report its state as "up", etc.

Some optimisations can be introduced in order to reduce the amount of time for the mentioned operations. These optimisations will improve the overall performance as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233598

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233655

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233904

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/233905

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233598
Committed: https://git.openstack.org/cgit/stackforge/networking-hyperv/commit/?id=1b6d3e9df001787f612a891a4544a2f24e57fd75
Submitter: Jenkins
Branch: master

commit 1b6d3e9df001787f612a891a4544a2f24e57fd75
Author: Claudiu Belu <email address hidden>
Date: Mon Oct 12 14:33:23 2015 +0300

    Caches VirtualSystemManagementService and MetricService

    Sets the mentioned attributes as object attributes, as they
    are constantly used and there is no need to instantiate them
    every time.

    Partial-Bug: #1505196

    Change-Id: Ieee7b9e587424f2a95aee85389adfaabf4032779

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233904
Committed: https://git.openstack.org/cgit/stackforge/networking-hyperv/commit/?id=2f4cbbb2002e5139c437978c75ea83d995967b41
Submitter: Jenkins
Branch: master

commit 2f4cbbb2002e5139c437978c75ea83d995967b41
Author: Claudiu Belu <email address hidden>
Date: Tue Oct 13 01:48:33 2015 +0300

    Avoids rebinding the same security group rules

    Previously, logic has been introduced to ensure that all the given
    security group rules have been applied. No need to force the rebinding
    after the agent restarts.

    Partial-Bug: #1505196

    Change-Id: I355629ddb78441eaf2aadfd27efc6dd1f1ce4de9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233655
Committed: https://git.openstack.org/cgit/stackforge/networking-hyperv/commit/?id=2abeee743ca02b1263bf7fdea36509bab52b23d8
Submitter: Jenkins
Branch: master

commit 2abeee743ca02b1263bf7fdea36509bab52b23d8
Author: Claudiu Belu <email address hidden>
Date: Mon Oct 12 15:58:23 2015 +0300

    Caches switch port, VLAN, VSID setting data WMI objects

    The Hyper-V Neutron Agent, during its first loop, will iterate
    over all the switch ports, checking their proper setup. In order
    to avoid wasting time querying the necessary WMI objects for each
    port, they are cached first.

    Initialises the cache for the mentioned objects. Thus, they will be
    retrieved from the cache while processing 'foo' port.

    This will also improve the general performance of the agent.

    Partial-Bug: #1505196

    Change-Id: I5a70600ffdb1aa29d113972f13485bbbb3138a47

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/233905
Committed: https://git.openstack.org/cgit/stackforge/networking-hyperv/commit/?id=b14a331637a43a7230d982683b4ab12f3eaf401c
Submitter: Jenkins
Branch: master

commit b14a331637a43a7230d982683b4ab12f3eaf401c
Author: Claudiu Belu <email address hidden>
Date: Tue Oct 13 02:03:52 2015 +0300

    Caches Security Group Rule ACLs

    In order to avoid querying for the mentioned WMI objects, adds
    a cache for them, verifying the existence of the Security Group
    Rules using the cache instead.

    Adds the Weight field to the SecurityGroupRuleR2 in order for the
    objects to have the same fields as the ACLs.

    Partial-Bug: #1505196

    Change-Id: I63d5984f3f9343e32e065f219567c4a821eb997b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/248157

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/248157
Committed: https://git.openstack.org/cgit/openstack/networking-hyperv/commit/?id=48eda2f40a6e115847ca996db764f5fd628521b0
Submitter: Jenkins
Branch: master

commit 48eda2f40a6e115847ca996db764f5fd628521b0
Author: Claudiu Belu <email address hidden>
Date: Fri Nov 20 18:16:00 2015 +0200

    Caches Hyper-V vSwitches

    Hyper-V vSwitches are queried for each port that is being bound.
    There is no need to query them each time, caching them can
    significantly improve performance.

    Change-Id: Ibd82775f07d81599fa6d16638af351d2a70f5f4a
    Partial-Bug: #1505196

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253625

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253625
Committed: https://git.openstack.org/cgit/openstack/networking-hyperv/commit/?id=9f6241e0a63a2bd36392eeb3c997786d53119a39
Submitter: Jenkins
Branch: master

commit 9f6241e0a63a2bd36392eeb3c997786d53119a39
Author: Claudiu Belu <email address hidden>
Date: Fri Dec 4 19:13:00 2015 +0200

    Create a new object instead of querying the default one

    Currently, a WMI Setting Data (Port VLAN, Port Security) is created
    by querying the default object. Under large scale environments,
    this can lead to huge execution times, as the query takes longer to
    complete. This can lead to late port binding and missed DHCP
    requests, leaving the VM with no IP.

    A solution for this is to create the needed object (.new()) instead
    of querying the default one.

    Change-Id: Ib492afe5387373a40d964e76fa6b4b0ee3b5666d
    Partial-Bug: #1505196

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/281877

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268262
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=6952aea528924e6033c370cd7d75f0a27db96993
Submitter: Jenkins
Branch: master

commit 6952aea528924e6033c370cd7d75f0a27db96993
Author: Claudiu Belu <email address hidden>
Date: Fri Jan 15 19:49:22 2016 +0200

    Caches switch port, VLAN, VSID setting data WMI objects

    The Hyper-V Neutron Agent, during its first loop, will iterate
    over all the switch ports, checking their proper setup. In order
    to avoid wasting time querying the necessary WMI objects for each
    port, they are cached first.

    Initialises the cache for the mentioned objects. Thus, they will be
    retrieved from the cache while processing 'foo' port.

    This will also improve the general performance of the agent.

    (cherry picked from commit 2abeee743ca02b1263bf7fdea36509bab52b23d8)

    Partial-Bug: #1505196

    Change-Id: I5a70600ffdb1aa29d113972f13485bbbb3138a47

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277667
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=a9fbb74879f313798ad6108bc2e9c9174392279d
Submitter: Jenkins
Branch: master

commit a9fbb74879f313798ad6108bc2e9c9174392279d
Author: Claudiu Belu <email address hidden>
Date: Tue Feb 9 04:47:15 2016 +0200

    Caches Hyper-V vSwitches

    Hyper-V vSwitches are queried for each port that is being bound.
    There is no need to query them each time, caching them can
    significantly improve performance.

    (cherry picked from commit 48eda2f40a6e115847ca996db764f5fd628521b0)

    Change-Id: Ibd82775f07d81599fa6d16638af351d2a70f5f4a
    Partial-Bug: #1505196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/281877
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=c9b8284df3029d400a977d7cc9def4c0565cede7
Submitter: Jenkins
Branch: master

commit c9b8284df3029d400a977d7cc9def4c0565cede7
Author: Claudiu Belu <email address hidden>
Date: Thu Feb 18 15:59:36 2016 +0200

    Create a new object instead of querying the default one

    Currently, a WMI Setting Data (Port VLAN, Port Security) is created
    by querying the default object. Under large scale environments,
    this can lead to huge execution times, as the query takes longer to
    complete. This can lead to late port binding and missed DHCP
    requests, leaving the VM with no IP.

    A solution for this is to create the needed object (.new()) instead
    of querying the default one.

    (cherry picked from commit 9f6241e0a63a2bd36392eeb3c997786d53119a39)

    Change-Id: Ib492afe5387373a40d964e76fa6b4b0ee3b5666d
    Partial-Bug: #1505196

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277668
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=5468b6f611575a147fb7db751d7d66e141da927d
Submitter: Jenkins
Branch: master

commit 5468b6f611575a147fb7db751d7d66e141da927d
Author: Claudiu Belu <email address hidden>
Date: Tue Feb 9 04:58:17 2016 +0200

    Caches Security Group Rule ACLs

    In order to avoid querying for the mentioned WMI objects, adds
    a cache for them, verifying the existence of the Security Group
    Rules using the cache instead.

    Adds the Weight field to the SecurityGroupRuleR2 in order for the
    objects to have the same fields as the ACLs.

    (cherry picked from commit b14a331637a43a7230d982683b4ab12f3eaf401c)

    Partial-Bug: #1505196

    Change-Id: I63d5984f3f9343e32e065f219567c4a821eb997b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268144
Committed: https://git.openstack.org/cgit/openstack/os-win/commit/?id=accb5214e5c59906083ba3144f109b065c3b32c7
Submitter: Jenkins
Branch: master

commit accb5214e5c59906083ba3144f109b065c3b32c7
Author: Claudiu Belu <email address hidden>
Date: Fri Jan 15 15:52:43 2016 +0200

    Creates baseutils module

    Creates BaseUtils class, which contains the _get_wmi_obj and
    _get_wmi_conn methods, commonly used in most of the utils
    classes.

    Caches the WMI connections, as they do not have to be unique.

    Sets the WMI _conn object, the Msvm_VirtualSystemManagementService
    as object attributes, as they are constantly used and there is no
    need to instantiate them every time.

    Partial-Bug: #1505196

    Change-Id: Ieee7b9e587424f2a95aee85389adfaabf4032779