EMC-POC: Nexus plugin: with multiple neutron processes, cannot kickstart config replay once it reaches the retry threshold.

[Danny Choi]

Replay parameters:
    - switch_heartbeat_time = 30
    - switch_replay_count = 3

Neutron worker parameters:
    - api_workers=4
    - rpc_workers=3

When it hits the replay count (3 in this case), the replay will stop and discontinue to pull the Nexus switch.

With both api and rpc workers set to zero, a VLAN delete or create event will kickstart the replay code again.

However, this does not work with multiple workers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/209597

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209597
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=e6673d534cdff2d5c8434ef9928a766b4304b003
Submitter: Jenkins
Branch: master

commit e6673d534cdff2d5c8434ef9928a766b4304b003
Author: Carol Bouchard <email address hidden>
Date: Wed Aug 5 12:36:16 2015 -0400

    Replace retry count with replay failure stats

    This code-set eliminates checking switch config replay attempts against
    max configurable attempts. Instead it will continue to contact the
    switch forever.

    I've repurposed some of the logic to provide statistics to
    identify if there are issues in replay process. There are two
    statistics: 1) contact 2) config. The contact stat identifies
    a problem contacting the switch. The operator would need to
    see if there is something wrong with the nexus switch or the path
    to the switch. The config stat identifies an issues with the config
    set we're trying to send to the switch. This latter case may require
    part of the config to be removed.

    Change-Id: Icb15d0b2d97f1c1c9f583989823a4dae256f3e65
    Closes-bug: #1469839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/213327

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/213327
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=a8c4bd753ba254b062612c1bcd85000656ebfa44
Submitter: Jenkins
Branch: stable/kilo

commit a8c4bd753ba254b062612c1bcd85000656ebfa44
Author: Carol Bouchard <email address hidden>
Date: Wed Aug 5 12:36:16 2015 -0400

    Replace retry count with replay failure stats

    This code-set eliminates checking switch config replay attempts against
    max configurable attempts. Instead it will continue to contact the
    switch forever.

    I've repurposed some of the logic to provide statistics to
    identify if there are issues in replay process. There are two
    statistics: 1) contact 2) config. The contact stat identifies
    a problem contacting the switch. The operator would need to
    see if there is something wrong with the nexus switch or the path
    to the switch. The config stat identifies an issues with the config
    set we're trying to send to the switch. This latter case may require
    part of the config to be removed.

    Change-Id: Icb15d0b2d97f1c1c9f583989823a4dae256f3e65
    Closes-bug: #1469839
    (cherry picked from commit e6673d534cdff2d5c8434ef9928a766b4304b003)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/246547

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/246547
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=7b1eb2b6d5e55563c084f60e44adc1d32706eb17
Submitter: Jenkins
Branch: master

commit d9b9a6421d7ff92e920ed21b01ebc7bf49e38bd6
Author: Sam Betts <email address hidden>
Date: Tue Sep 29 09:18:10 2015 +0100

    Set default branch for stable/kilo

    Change-Id: I31f51ff60f95639f459839f4c7d929d5ec7c458d

commit f08fb31f20c2d8cc1e6b71784cdfd9604895e16d
Author: Rich Curran <email address hidden>
Date: Thu Sep 3 13:23:52 2015 -0400

    ML2 cisco_nexus MD: VLAN not created on switch

    As described in DE588,
    "With neutron multiworkers configured, there is a potential race condition
    issue where some of the VLANs will not be configured on one or more N9k
    switches.

    /etc/neutron/neutron.conf
    -------------------------
    api_workers=3
    rpc_workers=3"

    Fix is to allow the vlan create command to be sent down to a switch
    under most event conditions. Long term fix will be to introduce a new
    column in the port binding DB table that indicates the true state of the
    entry/row.

    Closes-Bug: #1491940
    Change-Id: If1da1fcf16a450c1a4107da9970b18fc64936896
    (cherry picked from commit 0e48a16e77fc5ec5fd485a85f97f3650126fb6fe)

commit d400749e43e9d5a1fc92683b40159afce81edc95
Author: Carol Bouchard <email address hidden>
Date: Thu Sep 3 15:19:48 2015 -0400

    Create knob to prevent caching ssh connection

    Create a new initialization knob named never_cache_ssh_connection.
    This boolean is False by default allowing multiple ssh connections
    to the Nexus switch to be cached as it behaves today. When there
    are multiple neutron processes/controllers and/or non-neutron ssh(xml)
    connections, this is an issue since processes hold onto a connection
    while the Nexus devices supports a maximum of 8 sessions. As a result,
    further ssh connections will fail. In this case, the boolean should be
    set to True causing each connection to be closed when a neutron event
    is complete.

    Change-Id: I61ec303856b757dd8d9d43110fec8e7844ab7c6d
    Closes-bug: #1491108
    (cherry picked from commit 23551a4198c61e2e25a6382f27d47b0665f054b8)

commit 0050ea7f1fb3c22214d7ca49cfe641da86123e2c
Author: Carol Bouchard <email address hidden>
Date: Wed Sep 2 11:10:42 2015 -0400

    Bubble up exceptions when Nexus replay enabled

    There are several changes made surrounding this bug.

    1) When replay is enabled, we should bubble exceptions
       for received port create/update/delete post_commit
       transactions. This was suppressed earlier by
       1422738.

    2) When an exception is encountered during a
       post_commit transaction, the driver will no longer
       mark the switch state to inactive to force a replay.
       This is no longer needed since 1481856 was introduced.
       So from this point on, only the replay thread will
       determine the state of the connection to the switch.

    3) In addition to accommodating 1 & 2 above, more detail
       data verification was added to the test code.

    Change-Id: I97...

[Timothy Swanson (tiswanso)]

Corresponding tripleo-heat-templates and puppet-neutron bug: https://bugs.launchpad.net/tripleo/+bug/1551387