UCSM Driver does not check if desired vlan profile already exists

Bug #1470568 reported by Nikolay Fedotov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-cisco
Fix Released
Critical
Sandhya Dasu
Kilo
Fix Committed
Critical
Sandhya Dasu
Liberty
Fix Released
Critical
Sandhya Dasu

Bug Description

The driver tries to create vlan profile each time the update_port_postcommit is called. On a second+ time it fails with exception (see below). An instance goes in ERROR state (reason: No valid hosts).

2015-07-01 13:26:24.466 ^[[00;32mDEBUG networking_cisco.plugins.ml2.drivers.cisco.ucsm.ucsm_network_driver [^[[01;36mreq-df46c5ea-7edb-426e-b3ab-535d6ad852fd ^[[00;36mneutron f50b9b6e20f948b0ac72ae158e85591a^[[00;32m] ^[[01;35m^[[00;32mUCS Manager network driver created Vlan Profile OS-150 at fabric/lan/net-OS-150^[[00m ^[[00;33mfrom (pid=30211) _create_vlanprofile /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/ucsm/ucsm_network_driver.py:188^[[00m
2015-07-01 13:26:24.742 ^[[00;32mDEBUG networking_cisco.plugins.ml2.drivers.cisco.ucsm.ucsm_network_driver [^[[01;36mreq-df46c5ea-7edb-426e-b3ab-535d6ad852fd ^[[00;36mneutron f50b9b6e20f948b0ac72ae158e85591a^[[00;32m] ^[[01;35m^[[00;32mUCS Manager network driver found an entry that already exists os_1^[[00m ^[[00;33mfrom (pid=30211) update_serviceprofile /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/ucsm/ucsm_network_driver.py:424^[[00m

Steps to reproduce:
1. Launch instance attached to network1
2. Launch second instance attached to network1

Excepted behavior:
1. UCSM Driver creates vlan profile
2. UCSM Driver does create vlan profile because it already exists

Current behavior:

Revision history for this message
Nikolay Fedotov (nfedotov) wrote :

Sorry. The cause of the issue that I faced with is slightly different. I need to retest.

Changed in networking-cisco:
status: New → Incomplete
Revision history for this message
Nikolay Fedotov (nfedotov) wrote :

Able to reproduce it with SR-IOV ports.
Steps:
1. neutron net-create my-private-network
2. Create network with DHCP enabled (enabled by default): neutron subnet-create --ip_version 4 --gateway 10.0.6.1 --name my-private-subnet my-private-network 10.0.6.0/24
3. Create "direct" port: neutron port-create --vnic-type direct my-private-network
4. Boot an instance: nova boot --flavor m1.nano --image cirros-0.3.4-x86_64-uec --nic port-id=<port_id> test

Result:
* There is VLAN profile and VLAN added to vNIC
* There is no port profile created.
* In q-svc log you can find "... Manager network driver found an entry that already exists ..."

Revision history for this message
Nikolay Fedotov (nfedotov) wrote :

Able to reproduce it on a multi-node deployment with non-SR-IOV ports
Steps:
1. neutron net-create my-private-network
2. neutron subnet-create --ip_version 4 --gateway 10.0.6.1 --name my-private-subnet my-private-network 10.0.6.0/24
3. Create port1 on a compute1: neutron port-create 5f109df7-e71c-45c2-b3aa-d1e36a69d3d3 --binding:host_id=<hostname of compute1>
4. Create port2 on a compute2: neutron port-create 5f109df7-e71c-45c2-b3aa-d1e36a69d3d3 --binding:host_id=<hostname of compute2>
5. Boot an instance attached to port1
6. Boot an instance attached to port2

Result: VLAN is added to vNICs of an only first compute. The second compute does not have VLAN added to vNICs
Message in q-svc.log:
2015-07-15 08:20:30.368 DEBUG networking_cisco.plugins.ml2.drivers.cisco.ucsm.ucsm_network_driver [req-e62f2942-9150-40fa-b2ea-1f3e98e9e9c2 None None] UCS Manager network driver found an entry that already exists os3 from (pid=35642) update_serviceprofile /usr/local/lib/python2.7/distpackages/networking_cisco/plugins/ml2/drivers/cisco/ucsm/ucsm_network_driver.py:424

Changed in networking-cisco:
status: Incomplete → New
Sandhya Dasu (sadasu)
Changed in networking-cisco:
assignee: nobody → Sandhya Dasu (sadasu)
Changed in networking-cisco:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-cisco (master)

Reviewed: https://review.openstack.org/200304
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=4c8d9adb552b3c88deccac8f3289f622a2d7445e
Submitter: Jenkins
Branch: master

commit 4c8d9adb552b3c88deccac8f3289f622a2d7445e
Author: Sandhya Dasu <email address hidden>
Date: Thu Jul 9 19:01:52 2015 -0400

    Cisco UCSM driver updates to handle duplicate creations

    Each create, delete or update operation on the UCS Manager is part of its own
    transaction to handle duplicate creations on UCS Manager better. So, when a
    create operation, tries to create something that already exists, the plugin
    correctly ignores the duplicate exception and proceeds to the next step
    (instead of skipping it).

    Change-Id: I8f650dc8d656ac8ee4a839420b68423ad65123eb
    Closes-Bug: #1464500
    Closes-Bug: #1470568

Changed in networking-cisco:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-cisco (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/215253

Henry Gessau (gessau)
Changed in networking-cisco:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-cisco (stable/kilo)

Reviewed: https://review.openstack.org/215253
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=178f40f2a43192687188661d5fcedf394321e191
Submitter: Jenkins
Branch: stable/kilo

commit 178f40f2a43192687188661d5fcedf394321e191
Author: Sandhya Dasu <email address hidden>
Date: Thu Jul 9 19:01:52 2015 -0400

    Cisco UCSM driver updates to handle duplicate creations

    Each create, delete or update operation on the UCS Manager is part of its own
    transaction to handle duplicate creations on UCS Manager better. So, when a
    create operation, tries to create something that already exists, the plugin
    correctly ignores the duplicate exception and proceeds to the next step
    (instead of skipping it).

    Change-Id: I8f650dc8d656ac8ee4a839420b68423ad65123eb
    Closes-Bug: #1464500
    Closes-Bug: #1470568
    (cherry picked from commit 4c8d9adb552b3c88deccac8f3289f622a2d7445e)

tags: added: in-stable-kilo
Sam Betts (sambetts)
Changed in networking-cisco:
milestone: none → 1.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-cisco (master)

Fix proposed to branch: master
Review: https://review.openstack.org/246547

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-cisco (master)
Download full text (23.9 KiB)

Reviewed: https://review.openstack.org/246547
Committed: https://git.openstack.org/cgit/openstack/networking-cisco/commit/?id=7b1eb2b6d5e55563c084f60e44adc1d32706eb17
Submitter: Jenkins
Branch: master

commit d9b9a6421d7ff92e920ed21b01ebc7bf49e38bd6
Author: Sam Betts <email address hidden>
Date: Tue Sep 29 09:18:10 2015 +0100

    Set default branch for stable/kilo

    Change-Id: I31f51ff60f95639f459839f4c7d929d5ec7c458d

commit f08fb31f20c2d8cc1e6b71784cdfd9604895e16d
Author: Rich Curran <email address hidden>
Date: Thu Sep 3 13:23:52 2015 -0400

    ML2 cisco_nexus MD: VLAN not created on switch

    As described in DE588,
    "With neutron multiworkers configured, there is a potential race condition
    issue where some of the VLANs will not be configured on one or more N9k
    switches.

    /etc/neutron/neutron.conf
    -------------------------
    api_workers=3
    rpc_workers=3"

    Fix is to allow the vlan create command to be sent down to a switch
    under most event conditions. Long term fix will be to introduce a new
    column in the port binding DB table that indicates the true state of the
    entry/row.

    Closes-Bug: #1491940
    Change-Id: If1da1fcf16a450c1a4107da9970b18fc64936896
    (cherry picked from commit 0e48a16e77fc5ec5fd485a85f97f3650126fb6fe)

commit d400749e43e9d5a1fc92683b40159afce81edc95
Author: Carol Bouchard <email address hidden>
Date: Thu Sep 3 15:19:48 2015 -0400

    Create knob to prevent caching ssh connection

    Create a new initialization knob named never_cache_ssh_connection.
    This boolean is False by default allowing multiple ssh connections
    to the Nexus switch to be cached as it behaves today. When there
    are multiple neutron processes/controllers and/or non-neutron ssh(xml)
    connections, this is an issue since processes hold onto a connection
    while the Nexus devices supports a maximum of 8 sessions. As a result,
    further ssh connections will fail. In this case, the boolean should be
    set to True causing each connection to be closed when a neutron event
    is complete.

    Change-Id: I61ec303856b757dd8d9d43110fec8e7844ab7c6d
    Closes-bug: #1491108
    (cherry picked from commit 23551a4198c61e2e25a6382f27d47b0665f054b8)

commit 0050ea7f1fb3c22214d7ca49cfe641da86123e2c
Author: Carol Bouchard <email address hidden>
Date: Wed Sep 2 11:10:42 2015 -0400

    Bubble up exceptions when Nexus replay enabled

    There are several changes made surrounding this bug.

    1) When replay is enabled, we should bubble exceptions
       for received port create/update/delete post_commit
       transactions. This was suppressed earlier by
       1422738.

    2) When an exception is encountered during a
       post_commit transaction, the driver will no longer
       mark the switch state to inactive to force a replay.
       This is no longer needed since 1481856 was introduced.
       So from this point on, only the replay thread will
       determine the state of the connection to the switch.

    3) In addition to accommodating 1 & 2 above, more detail
       data verification was added to the test code.

    Change-Id: I97...

Changed in networking-cisco:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.