NetApp drivers fails to create share_servers on controllers running ONTAP 8.3

[Ben Swartzlander]

2015-02-25 16:22:34.323 DEBUG oslo_concurrency.lockutils [req-8e331d05-91e4-4ea1-aed8-5014b815a950 7010f9db730247d8aabd0e67db453679 81b9bdfd56f14f93866d1e364ad10c8f] Lock "share_manager_3815eea1-d45a-4308-9723-5183afc1ea69" released by "_provide_share_server_for_share" :: held 28.735s from (pid=14112) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:442
2015-02-25 16:22:34.324 ERROR manila.share.manager [req-8e331d05-91e4-4ea1-aed8-5014b815a950 7010f9db730247d8aabd0e67db453679 81b9bdfd56f14f93866d1e364ad10c8f] Failed to get share server for share creation.
2015-02-25 16:22:34.349 DEBUG oslo_concurrency.lockutils [req-36afb752-fef3-45a1-8fa7-e64f1409b68d 7010f9db730247d8aabd0e67db453679 81b9bdfd56f14f93866d1e364ad10c8f] Lock "share_manager_3815eea1-d45a-4308-9723-5183afc1ea69" acquired by "_provide_share_server_for_share" :: waited 26.364s from (pid=14112) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:430
2015-02-25 16:22:34.363 ERROR oslo_messaging.rpc.dispatcher [req-8e331d05-91e4-4ea1-aed8-5014b815a950 7010f9db730247d8aabd0e67db453679 81b9bdfd56f14f93866d1e364ad10c8f] Exception during message handling: NetApp API failed. Reason - 13001:Port "e0a-1003" on node "bsupreme-cluster-01" is not a member of a broadcast domain.
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 272, in create_share
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher {'status': 'error'})
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 266, in create_share
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher context, share_network_id, share_id)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 217, in _provide_share_server_for_share
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher return _provide_share_server_for_share()
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 431, in inner
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 209, in _provide_share_server_for_share
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher share_server = self._setup_server(context, share_server)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 518, in _setup_server
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/manager.py", line 491, in _setup_server
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher metadata=metadata)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/driver.py", line 313, in setup_server
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 80, in _setup_server
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher return self.library.setup_server(network_info, metadata)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 85, in trace_wrapper
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 291, in setup_server
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher vserver_name = self._create_vserver_if_nonexistent(network_info)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 85, in trace_wrapper
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 325, in _create_vserver_if_nonexistent
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher self._client.delete_vserver(vserver_name, vserver_client)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 321, in _create_vserver_if_nonexistent
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher network_info)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 85, in trace_wrapper
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 355, in _create_vserver_lifs
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher vserver_client)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 85, in trace_wrapper
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 366, in _create_lif_if_nonexistent
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher self.configuration.netapp_lif_name_template)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/utils.py", line 85, in trace_wrapper
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 324, in create_network_interface
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher self.send_request('net-interface-create', api_args)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/client/client_base.py", line 60, in send_request
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher return self.connection.invoke_successfully(request, enable_tunneling)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/manila/manila/share/drivers/netapp/dataontap/client/api.py", line 258, in invoke_successfully
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher raise NaApiError(code, msg)
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher NaApiError: NetApp API failed. Reason - 13001:Port "e0a-1003" on node "bsupreme-cluster-01" is not a member of a broadcast domain.
2015-02-25 16:22:34.363 TRACE oslo_messaging.rpc.dispatcher

Initial investigations shows that Ontap 8.3 requires an ipspace (or "broadcast domain") for LIF creation and the NetApp driver doesn't provide that.

[Valeriy Ponomaryov (vponomaryov)]

Just reminder, tthat fix should be done for stable/juno version too.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/168534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168534
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=eec09463db28e4ba21324c206c6426145b92399a
Submitter: Jenkins
Branch: master

commit eec09463db28e4ba21324c206c6426145b92399a
Author: Clinton Knight <email address hidden>
Date: Wed Mar 18 22:11:33 2015 -0400

    NetApp cDOT multi-SVM driver should not start with cDOT 8.3

    The multi-SVM cDOT driver that ships with Kilo does not support the
    8.3 cDOT release. The driver should fail fast if 8.3 is detected.

    Change-Id: Iac7bafd101f0b0ad190a49cd2bba79bd25dee028
    Related-Bug: #1425754
    Closes-Bug: #1433872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/196184
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=dab1e3bc14553d978862c47777fa4441e17e1a05
Submitter: Jenkins
Branch: master

commit dab1e3bc14553d978862c47777fa4441e17e1a05
Author: Clinton Knight <email address hidden>
Date: Fri Mar 20 17:31:42 2015 -0400

    Enable Manila multi-SVM driver on NetApp cDOT 8.3

    cDOT 8.3 includes numerous new networking features, at least one
    of which (broadcast domains) must be used when creating network
    interfaces. This patch enables the multi-SVM driver on cDOT 8.3
    by ensuring that each physical port selected to host share server
    logical interfaces (LIFs) is a member of a broadcast domain.
    A couple of other API tweaks are included that also impeded
    operation on cDOT 8.3.

    Closes-Bug: #1425754
    Change-Id: Ibe92aee217462bce06485e195b930f2fdf43fcc0