NetApp: Manila cDOT multi-svm driver cannot handle duplicate ip addresses

[Andrew Kerr]

When running tempest tests against the manila multi-svm driver, even with single threaded concurrency, the driver will often fail due to a duplicate ip address issue on the cluster. This seems to be due to multiple tenants all using the same IP space, which is perfectly legal and legitimate use of neutron. It is also causing failures in our CI system. An easy way to reproduce the issue is to run tempest against the multi-svm driver at higher concurrency, such as 4 or 6. An example of the traceback is below:

2015-09-02 03:53:01.924 8140 ERROR manila.share.manager [req-17889b4b-8cd8-4fd6-aa25-222f3ab024f5 ] Failed to get share server for consistency group creation.
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher [req-17889b4b-8cd8-4fd6-aa25-222f3ab024f5 ] Exception during message handling: Duplicate ip address 10.100.0.3
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 1217, in create_consistency_group
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher context, cg_id, {'status': constants.STATUS_ERROR})
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 1210, in create_consistency_group
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher context, share_network_id, group_ref, cgsnapshot=snap_ref
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 444, in _provide_share_server_for_cg
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher return _provide_share_server_for_cg()
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 436, in _provide_share_server_for_cg
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher context, compatible_share_server)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 983, in _setup_server
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher self.driver.deallocate_network(context, share_server['id'])
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 934, in _setup_server
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher network_info, metadata=metadata)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/driver.py", line 377, in setup_server
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher return self._setup_server(*args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 116, in _setup_server
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher return self.library.setup_server(network_info, metadata)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 87, in trace_wrapper
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 114, in setup_server
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher self._create_vserver_if_nonexistent(vserver_name, network_info)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 87, in trace_wrapper
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 157, in _create_vserver_if_nonexistent
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher self._client.delete_vserver(vserver_name, vserver_client)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 153, in _create_vserver_if_nonexistent
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher network_info)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 87, in trace_wrapper
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 186, in _create_vserver_lifs
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher vserver_client)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 87, in trace_wrapper
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_multi_svm.py", line 209, in _create_lif_if_nonexistent
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher self.configuration.netapp_lif_name_template)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 87, in trace_wrapper
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher result = f(self, *args, **kwargs)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 397, in create_network_interface
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher self.send_request('net-interface-create', api_args)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_base.py", line 63, in send_request
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher return self.connection.invoke_successfully(request, enable_tunneling)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/netapp_lib/api/zapi/zapi.py", line 291, in invoke_successfully
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher raise NaApiError(code, msg)
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher NaApiError: NetApp API failed. Reason - 48:Duplicate ip address 10.100.0.3
2015-09-02 03:53:01.939 8140 ERROR oslo_messaging.rpc.dispatcher

[Ben Swartzlander (bswartz)]

Although this is of high importance, there's no proposed fix so we must push this one out.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/251487

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/251487
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=6369e3f9c87b7815fdf5f826d42f861b0160affd
Submitter: Jenkins
Branch: master

commit 6369e3f9c87b7815fdf5f826d42f861b0160affd
Author: Clinton Knight <email address hidden>
Date: Sun Sep 13 12:22:08 2015 -0400

    NetApp cDOT multi-SVM driver can't handle duplicate addresses

    When operating in DHSS==True mode, the cDOT driver can receive
    separate Neutron subnets for different tenants with overlapping
    IP address ranges. This is typically seen in Tempest testing,
    and it is likely to occur for production clouds as well. VLAN
    segmentation is insufficient by itself to resolve the problem.
    Starting with 8.3, clustered Data ONTAP supports IPspaces,
    which allow different storage virtual machines (i.e. share
    servers) to utilize overlapping address spaces. This commit
    resolves the issue on cDOT 8.3 by creating a new IPspace for
    each unique tenant network segment (i.e., each Neutron subnet)
    it encounters while creating share servers. It also includes
    some minor improvements to error handling when vservers aren't
    created and configured successfully plus some trivial refactoring
    in the vserver creation path.

    Closes-Bug: #1491384
    Change-Id: I808547c7d38e6bb61ae10ecdb174d3ab52b1443a

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/manila 2.0.0.0b1 development milestone.