NetApp driver doesn't check for valid license before reporting "revert_to_snapshot_support"

[Goutham Pacha Ravi]

The driver claims to the scheduler that revert_to_snapshot_support, a user visible extra-spec is always supported. However, when the license is not applied, the back end cannot support this feature. This puts the share in an invalid state. We should check for the license before advertising the capability.

Entering method restore_snapshot trace_wrapper /opt/stack/new/manila/manila/share/drivers/netapp/utils.py:85
2017-04-01 19:14:08.414 7024 DEBUG manila.share.drivers.netapp.dataontap.client.api [req-7f7dc1af-9b4a-4cb1-bfeb-290f7ecaff7f a0b3c19850df4cdc85815b70c5c289bc 675655c57ae04e14a86af8635402686f - - -] Request: <netapp xmlns="http://www.netapp.com/filer/admin" version="1.32" vfiler="os_b2220e7d-42c1-4f16-a5c7-67781cd691e3">
  <snapshot-restore-volume>
    <volume>share_18f15921_e62f_408d_a745_62e38be0f23e</volume>
    <snapshot>share_snapshot_f2dd60a8_f852_408a_af2f_3000de918b7d</snapshot>
  </snapshot-restore-volume>
</netapp>
 invoke_elem /opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py:229
2017-04-01 19:14:08.438 7024 DEBUG manila.share.drivers.netapp.dataontap.client.api [req-7f7dc1af-9b4a-4cb1-bfeb-290f7ecaff7f a0b3c19850df4cdc85815b70c5c289bc 675655c57ae04e14a86af8635402686f - - -] Response: <results xmlns="http://www.netapp.com/filer/admin" reason="You do not have a valid license for &quot;SnapRestore&quot;. Reason: Package &quot;SnapRestore&quot; is not licensed in the cluster." status="failed" errno="13001"/>
 invoke_elem /opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py:250
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager [req-7f7dc1af-9b4a-4cb1-bfeb-290f7ecaff7f a0b3c19850df4cdc85815b70c5c289bc 675655c57ae04e14a86af8635402686f - - -] Share 13799954-48df-428a-a2a7-7f58f53a7179 could not be reverted to snapshot e79e5535-3dd9-41ba-8d70-4457d829822d.
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager Traceback (most recent call last):
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/manager.py", line 2501, in _revert_to_snapshot
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager share_server=share_server)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 59, in revert_to_snapshot
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager return self.library.revert_to_snapshot(context, snapshot, **kwargs)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 803, in revert_to_snapshot
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager vserver_client.restore_snapshot(share_name, snapshot_name)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 86, in trace_wrapper
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager result = f(self, *args, **kwargs)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 2195, in restore_snapshot
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager self.send_request('snapshot-restore-volume', api_args)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_base.py", line 84, in send_request
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager return self.connection.invoke_successfully(request, enable_tunneling)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py", line 274, in invoke_successfully
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager raise NaApiError(code, msg)
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager NaApiError: NetApp API failed. Reason - 13001:You do not have a valid license for "SnapRestore". Reason: Package "SnapRestore" is not licensed in the cluster.
2017-04-01 19:14:08.439 7024 ERROR manila.share.manager
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server [req-7f7dc1af-9b4a-4cb1-bfeb-290f7ecaff7f a0b3c19850df4cdc85815b70c5c289bc 675655c57ae04e14a86af8635402686f - - -] Exception during message handling
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 167, in wrapped
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 519, in wrapper
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2479, in revert_to_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server access_rules)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2520, in _revert_to_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server {'status': constants.STATUS_AVAILABLE})
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2501, in _revert_to_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server share_server=share_server)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/drv_multi_svm.py", line 59, in revert_to_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server return self.library.revert_to_snapshot(context, snapshot, **kwargs)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 803, in revert_to_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server vserver_client.restore_snapshot(share_name, snapshot_name)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/utils.py", line 86, in trace_wrapper
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 2195, in restore_snapshot
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server self.send_request('snapshot-restore-volume', api_args)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/client_base.py", line 84, in send_request
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server return self.connection.invoke_successfully(request, enable_tunneling)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/netapp/dataontap/client/api.py", line 274, in invoke_successfully
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server raise NaApiError(code, msg)
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server NaApiError: NetApp API failed. Reason - 13001:You do not have a valid license for "SnapRestore". Reason: Package "SnapRestore" is not licensed in the cluster.
2017-04-01 19:14:08.663 7024 ERROR oslo_messaging.rpc.server
2017-04-01 19:14:12.990 7024 DEBUG oslo_messaging._drivers.amqpdriver [-] received message with unique_id: e5dd6fa629c4440bb4601bb3e1051aa6 __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:215

[Jason Grosso (jgrosso)]

Hey Naresh any updates for this defect, or any help need from the upstream community?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/665142

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665142
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f333b88e2521c8a1fecb88206cfb08046957346f
Submitter: Zuul
Branch: master

commit f333b88e2521c8a1fecb88206cfb08046957346f
Author: Naresh Kumar Gunjalli <email address hidden>
Date: Mon Jan 21 11:36:44 2019 -0500

    Check NetApp SnapRestore license for pools

    Added check to validate SnapRestore license for
    NetApp ONTAP pools before setting revert_to_snapshot_support
    to true/false for the respective pool.
    This was always set to True before.

    Change-Id: I4750d28c8acfe6d68a91e129e767c6e8607e6a75
    Closes-Bug: #1678524

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/669759

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669759
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=9e39a138fbe8c0546898543de8c007238caf0ef0
Submitter: Zuul
Branch: stable/stein

commit 9e39a138fbe8c0546898543de8c007238caf0ef0
Author: Naresh Kumar Gunjalli <email address hidden>
Date: Mon Jan 21 11:36:44 2019 -0500

    Check NetApp SnapRestore license for pools

    Added check to validate SnapRestore license for
    NetApp ONTAP pools before setting revert_to_snapshot_support
    to true/false for the respective pool.
    This was always set to True before.

    Change-Id: I4750d28c8acfe6d68a91e129e767c6e8607e6a75
    Closes-Bug: #1678524
    (cherry picked from commit f333b88e2521c8a1fecb88206cfb08046957346f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/686231

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686231
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=62996ad8f663e45758227b927058d80bab3758df
Submitter: Zuul
Branch: stable/rocky

commit 62996ad8f663e45758227b927058d80bab3758df
Author: Naresh Kumar Gunjalli <email address hidden>
Date: Mon Jan 21 11:36:44 2019 -0500

    Check NetApp SnapRestore license for pools

    Added check to validate SnapRestore license for
    NetApp ONTAP pools before setting revert_to_snapshot_support
    to true/false for the respective pool.
    This was always set to True before.

    Change-Id: I4750d28c8acfe6d68a91e129e767c6e8607e6a75
    Closes-Bug: #1678524
    (cherry picked from commit f333b88e2521c8a1fecb88206cfb08046957346f)
    (cherry picked from commit 9e39a138fbe8c0546898543de8c007238caf0ef0)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/687958

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687958
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=f581e987a5bf0bbdaa2154233d8a94240b6f8da1
Submitter: Zuul
Branch: stable/queens

commit f581e987a5bf0bbdaa2154233d8a94240b6f8da1
Author: Naresh Kumar Gunjalli <email address hidden>
Date: Mon Jan 21 11:36:44 2019 -0500

    Check NetApp SnapRestore license for pools

    Added check to validate SnapRestore license for
    NetApp ONTAP pools before setting revert_to_snapshot_support
    to true/false for the respective pool.
    This was always set to True before.

    Change-Id: I4750d28c8acfe6d68a91e129e767c6e8607e6a75
    Closes-Bug: #1678524
    (cherry picked from commit f333b88e2521c8a1fecb88206cfb08046957346f)
    (cherry picked from commit 9e39a138fbe8c0546898543de8c007238caf0ef0)
    (cherry picked from commit 62996ad8f663e45758227b927058d80bab3758df)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.3.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 7.4.0 release.