Netapp: DHSS=True share create fails trying to change broadcast domain on port

[Tom Barron]

Deploying with NetApp driver with DHSS=True backend the driver attempts to set up a new
vserver/SVM as a manila share server, and a new LIF (on each node in the NetApp cluster)
within that VM as export location for shares.

Share creates are failing because the vserver create fails:

2016-11-30 20:46:32.027 261185 ERROR manila.share.drivers.netapp.dataontap.cluster_mode.lib_multi_svm [req-c0702498-210d-4f18-9b6c-79b96a954d66 d6fd2e259d4c4340a5a7ab7037dbc44a 3fa8f10886ca4661a0914ab09db07676 - - -] Failed to configure Vserver.

Vserver create fails because even though it is actually completed on the backend it's LIFs are not
created and set up, which failure in turn is due to one of the ports for the LIF being in use by
other LIFs when the driver attempts to change its "broadcast domain" from "Default" to "OpenStack":

 invoke_elem /usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/api.py:248
2016-11-30 20:46:31.996 261185 DEBUG manila.share.drivers.netapp.dataontap.client.api [req-c0702498-210d-4f18-9b6c-79b96a954d66 d6fd2e259d4c4340a5a7ab7037dbc44a 3fa8f10886ca4661a0914ab09db07676 - - -] Request: <netapp xmlns="http://www.netapp.com/filer/admin" version="1.100">
  <net-port-broadcast-domain-remove-ports>
    <broadcast-domain>Default</broadcast-domain>
    <ipspace>Default</ipspace>
    <ports>
      <net-qualified-port-name>ntap2554openstack-01:e0c</net-qualified-port-name>
    </ports>
  </net-port-broadcast-domain-remove-ports>
</netapp>
 invoke_elem /usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/api.py:227
2016-11-30 20:46:32.026 261185 DEBUG manila.share.drivers.netapp.dataontap.client.api [req-c0702498-210d-4f18-9b6c-79b96a954d66 d6fd2e259d4c4340a5a7ab7037dbc44a 3fa8f10886ca4661a0914ab09db07676 - - -] Response: <results xmlns="http://www.netapp.com/filer/admin" reason="Port &quot;ntap2554openstack-01:e0c&quot; cannot be used because it is currently the home port or current port of a LIF. To view LIFs with a specified home port, use the &quot;network interface show -home-node &lt;nodename&gt; -home-port {&lt;netport&gt;|&lt;ifgrp&gt;}&quot; command. To view LIFs with a specified current port, use the &quot;network interface show -curr-node &lt;nodename&gt; -curr-port {&lt;netport&gt;|&lt;ifgrp&gt;}&quot; command. " status="failed" errno="13001"/>
 invoke_elem /usr/lib/python2.7/site-packages/manila/share/drivers/netapp/dataontap/client/api.py:248

Note that multiple LIFs per port is normal on NetApps and at least in lab situations (and
perhaps elsewher) it will be a normal circumstance to use a NetApp with pre-existing data port
LIFs before OpenStack is run with the multi-svm driver.

A poor customer experience with an obscure failure will be the current result.

[Tom Barron (tpb)]

Log snippets above are from the share log. Here's a fuller pastebin of that log: http://paste.fedoraproject.org/494411/80540061/

[Tom Barron (tpb)]

Here's the relevant backend configuation:

    [tripleo_netapp]
    netapp_lif_name_template=os_%(net_allocation_id)s
    share_driver=manila.share.drivers.netapp.common.NetAppDriver
    netapp_vserver=tbarron_0
    netapp_password=netapp123
    netapp_storage_family=ontap_cluster
    driver_handles_share_servers=True
    share_backend_name=tripleo_netapp
    netapp_port_name_search_pattern=(.*)
    netapp_root_volume_aggregate=aggr1_n1
    netapp_trace_flags=network,api
    netapp_login=admin
    netapp_vserver_name_template=os_%s
    netapp_server_port=80
    netapp_root_volume_name=root
    netapp_volume_name_template=share_%(share_id)s
    netapp_aggregate_name_search_pattern=(.*)
    netapp_transport_type=http
    netapp_server_hostname=10.9.20.123
    network_api_class='manila.network.standalone_network_plugin.StandaloneNetworkPlugin'
    standalone_network_plugin_allowed_ip_ranges='192.168.1.200-192.168.1.220'
    standalone_network_plugin_ip_version=4
    standalone_network_plugin_mask=255.255.0.0
    standalone_network_plugin_gateway=192.168.14.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/406042

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/406042
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=fa32c3787e682f81d497164d3331a08dff03cb89
Submitter: Jenkins
Branch: master

commit fa32c3787e682f81d497164d3331a08dff03cb89
Author: Tom Barron <email address hidden>
Date: Fri Dec 2 05:16:28 2016 -0500

    NetApp: set proper broadcast domain for IPspace

    When DHSS=True, the cDOT driver creates vservers to act as
    manila share servers and sets up LIFs within the vservers
    as export locations using physical ports. The LIFs in the
    NetApp cluster may use overlapping addresses provided that
    they are put in separate IPspaces and the physical ports
    underneath them are put in corresponding broadcast domains.

    Fix the cDOT driver to maintain a proper 1-1 relation between
    IPSpaces and broadcast domains as documented here [1].

    [1] https://library.netapp.com/ecmdocs/ECMP1636021/html/GUID-CB9BD2E2-D085-446E-8EB2-6BF609E42ABA.html

    Change-Id: I32968e82cc679bb7efe492f3ca980e16e5e29c83
    Closes-bug: #1646603

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417226

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417226
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e1069b5c8ceb05aac1102f9b7d90b9ee51830e90
Submitter: Jenkins
Branch: stable/newton

commit e1069b5c8ceb05aac1102f9b7d90b9ee51830e90
Author: Tom Barron <email address hidden>
Date: Fri Dec 2 05:16:28 2016 -0500

    NetApp: set proper broadcast domain for IPspace

    When DHSS=True, the cDOT driver creates vservers to act as
    manila share servers and sets up LIFs within the vservers
    as export locations using physical ports. The LIFs in the
    NetApp cluster may use overlapping addresses provided that
    they are put in separate IPspaces and the physical ports
    underneath them are put in corresponding broadcast domains.

    Fix the cDOT driver to maintain a proper 1-1 relation between
    IPSpaces and broadcast domains as documented here [1].

    [1] https://library.netapp.com/ecmdocs/ECMP1636021/html/GUID-CB9BD2E2-D085-446E-8EB2-6BF609E42ABA.html

    Change-Id: I32968e82cc679bb7efe492f3ca980e16e5e29c83
    Closes-bug: #1646603
    (cherry picked from commit fa32c3787e682f81d497164d3331a08dff03cb89)