NetApp (DHSS = True) driver with Standalone Plugin throws error on share server deletion

[Yogesh]

When I am trying to delete the active Share Server created by the NetApp CIFS driver with DHSS = True on Standalone plugin, I am seeing following error on m-shr log - IndexError: list index out
of range.

The Share server goes from "active" to "error" state.

Unless, I manually delete the SVM from backend, I am not able to clean up the SS on manila side.

m-shr logs -

Jun 15 14:36:49.200042 openstack7 manila-share[14147]: 2017-06-15 14:36:49.185 DEBUG manila.share.drivers.netapp.utils [req-5e772116-5bdf-4836-98d1-ab03b9ebf99f bbf164e28864491587e27383f9ac1a6f
 00b62ce618d947f780266881fc4f238c] Leaving method get_network_interfaces from (pid=14176) trace_wrapper /opt/stack/manila/manila/share/drivers/netapp/utils.py:88
Jun 15 14:36:49.200246 openstack7 manila-share[14147]: 2017-06-15 14:36:49.186 ERROR manila.share.manager [req-5e772116-5bdf-4836-98d1-ab03b9ebf99f bbf164e28864491587e27383f9ac1a6f 00b62ce618d9
47f780266881fc4f238c] Share server '83bc8ed3-8e21-4cee-9541-42bd8f2a4f4a' failed on deletion.: IndexError: list index out of range
Jun 15 14:36:49.243874 openstack7 manila-share[14147]: 2017-06-15 14:36:49.235 DEBUG oslo_concurrency.lockutils [req-5e772116-5bdf-4836-98d1-ab03b9ebf99f bbf164e28864491587e27383f9ac1a6f 00b62c
e618d947f780266881fc4f238c] Lock "share_manager_64e702f2-1f1a-49af-b561-ea040a021ea5" released by "manila.share.manager._wrapped_delete_share_server" :: held 0.261s: IndexError: list index out
of range from (pid=14176) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282
Jun 15 14:36:49.247886 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 ERROR oslo_messaging.rpc.server [req-5e772116-5bdf-4836-98d1-ab03b9ebf99f bbf164e28864491587e27383f9ac1a6f 00b62ce
618d947f780266881fc4f238c] Exception during message handling: IndexError: list index out of range
Jun 15 14:36:49.248194 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
Jun 15 14:36:49.248414 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line
 153, in _process_incoming
Jun 15 14:36:49.248642 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 15 14:36:49.249009 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 213, in dispatch
Jun 15 14:36:49.249221 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 15 14:36:49.249566 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 183, in _do_dispatch
Jun 15 14:36:49.249894 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 15 14:36:49.250266 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 169, in wrapped
Jun 15 14:36:49.250584 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server return f(self, *args, **kwargs)
Jun 15 14:36:49.250892 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 548, in wrapper
Jun 15 14:36:49.251189 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server return func(self, *args, **kwargs)
Jun 15 14:36:49.251489 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3321, in delete_share_server
Jun 15 14:36:49.251741 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server _wrapped_delete_share_server()
Jun 15 14:36:49.252082 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
Jun 15 14:36:49.252408 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server return f(*args, **kwargs)
Jun 15 14:36:49.252725 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3317, in _wrapped_delete_share_server
Jun 15 14:36:49.253024 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server context, server_id, {'status': constants.STATUS_ERROR})
Jun 15 14:36:49.253319 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 15 14:36:49.253557 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server self.force_reraise()
Jun 15 14:36:49.253776 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 15 14:36:49.254035 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jun 15 14:36:49.254310 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3310, in _wrapped_delete_share_server
Jun 15 14:36:49.254527 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server security_services=security_services)
Jun 15 14:36:49.254748 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/driver.py", line 1038, in teardown_server
Jun 15 14:36:49.254951 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server return self._teardown_server(*args, **kwargs)
Jun 15 14:36:49.255243 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 139, in _teardown_server
Jun 15 14:36:49.255547 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server self.library.teardown_server(server_details, **kwargs)
Jun 15 14:36:49.255753 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 86, in trace_wrapper
Jun 15 14:36:49.255996 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Jun 15 14:36:49.256207 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 324, in teardown_server
Jun 15 14:36:49.256409 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server self._delete_vserver(vserver, security_services=security_services)
Jun 15 14:36:49.256721 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 86, in trace_wrapper
Jun 15 14:36:49.257081 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server result = f(self, *args, **kwargs)
Jun 15 14:36:49.257465 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 338, in _delete_vserver
Jun 15 14:36:49.257729 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server vlan = home_port.split('-')[1]
Jun 15 14:36:49.257940 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server IndexError: list index out of range
Jun 15 14:36:49.258262 openstack7 manila-share[14147]: 2017-06-15 14:36:49.244 TRACE oslo_messaging.rpc.server

[Goutham Pacha Ravi (gouthamr)]

Confirmed that this bug exists with NFS as well. Updated the topic appropriately. Tagged low because it's a vendor driver bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477691
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=abe3224774ebbad0aa22365b647a381efa28ec49
Submitter: Jenkins
Branch: master

commit abe3224774ebbad0aa22365b647a381efa28ec49
Author: Goutham Pacha Ravi <email address hidden>
Date: Sun Jun 18 03:45:42 2017 -0400

    NetApp cDOT: Fix share server deletion

    The DHSS=True cDOT driver can be used with non-segmented
    networks, and the driver should not assume that network
    interfaces are always assigned to VLAN ports.

    Change-Id: Ifffae0f5e42d16ea40d693116b720c7f7d57c971
    Closes-Bug: #1698250

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/478244

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/478244
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c3dbe188fc748729b7b846286decd5257a3a8c56
Submitter: Jenkins
Branch: stable/ocata

commit c3dbe188fc748729b7b846286decd5257a3a8c56
Author: Goutham Pacha Ravi <email address hidden>
Date: Sun Jun 18 03:45:42 2017 -0400

    NetApp cDOT: Fix share server deletion

    The DHSS=True cDOT driver can be used with non-segmented
    networks, and the driver should not assume that network
    interfaces are always assigned to VLAN ports.

    Change-Id: Ifffae0f5e42d16ea40d693116b720c7f7d57c971
    Closes-Bug: #1698250
    (cherry picked from commit abe3224774ebbad0aa22365b647a381efa28ec49)
    (cherry picked from commit 194c1152ab971966f35717428e743f011721ee61)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: driverfixes/newton
Review: https://review.openstack.org/483049

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 5.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483049
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=0e21145235f8ed642dedbfb3bd126b4f3060145d
Submitter: Zuul
Branch: driverfixes/newton

commit 0e21145235f8ed642dedbfb3bd126b4f3060145d
Author: Goutham Pacha Ravi <email address hidden>
Date: Sun Jun 18 03:45:42 2017 -0400

    NetApp cDOT: Fix share server deletion

    The DHSS=True cDOT driver can be used with non-segmented
    networks, and the driver should not assume that network
    interfaces are always assigned to VLAN ports.

    Change-Id: Ifffae0f5e42d16ea40d693116b720c7f7d57c971
    Closes-Bug: #1698250
    (cherry picked from commit abe3224774ebbad0aa22365b647a381efa28ec49)
    (cherry picked from commit 194c1152ab971966f35717428e743f011721ee61)
    (cherry picked from commit c3dbe188fc748729b7b846286decd5257a3a8c56)