NetApp NFS driver fails to startup

[Seth Forgosh]

Steps to reproduce:
1) Create a Vserver scoped user and configure the NetApp cDOT NFS driver per instructions here: http://netapp.github.io/openstack-deploy-ops-guide/ocata/content/cinder.fas.configuration.html#cinder.cdot.account_permissions
2) Start the cinder-volume service, it fails to start up.

Traceback:

ssh_known_hosts'. __init__ /usr/lib/python2.7/site-packages/cinder/ssh_utils.py:90
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode [req-943fc63e-d3e1-4814-a293-1c291ceb9141 - - - - -] Could not get performance base counter name. Performance-based scheduler functions may not be available.
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode Traceback (most recent call last):
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/performance/perf_cmode.py", line 47, in _init_counter_info
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode 'avg_processor_busy'))
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/performance/perf_base.py", line 224, in _get_base_counter_name
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode object_name, counter_name)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode return f(*args, **kwargs)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 289, in get_performance_counter_info
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode enable_tunneling=False)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode return f(*args, **kwargs)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 90, in send_request
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode return self.connection.invoke_successfully(request, enable_tunneling)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode raise NaApiError(code, msg)
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode NaApiError: NetApp API failed. Reason - 13001:Object "system:constituent" was not found.
2017-05-30 19:39:15.602 28704 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager [req-943fc63e-d3e1-4814-a293-1c291ceb9141 - - - - -] Failed to initialize driver.
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager Traceback (most recent call last):
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 432, in init_host
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager self.driver.check_for_setup_error()
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py", line 101, in check_for_setup_error
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager self.ssc_library.check_api_permissions()
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/capabilities.py", line 75, in check_api_permissions
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager api):
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 750, in check_cluster_api
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return self._check_cluster_api(object_name, operation_name, api)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 781, in _check_cluster_api
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager 'system-user-capability-get-iter', api_args, False)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return f(*args, **kwargs)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 90, in send_request
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager return self.connection.invoke_successfully(request, enable_tunneling)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager raise NaApiError(code, msg)
2017-05-30 19:39:15.672 28704 ERROR cinder.volume.manager NaApiError: NetApp API failed. Reason - 13003:Insufficient privileges: user 'test' does not have read access to this resource
packages/oslo_db/sqlalchemy/engines.py:261

show-user-capability ZAPI fails.

[sklgromek (t-gromowski)]

I think this may be related with this bug #1668261
In my environments i use Kolla with images centos-source-*
I use Iscsi protocol but got similar errors:

-06-08 09:19:08.983 7 INFO cinder.volume.manager [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Determined volume DB was not empty at startup.
2017-06-08 09:19:09.082 7 INFO cinder.volume.drivers.netapp.utils [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Could not run rpm command: Unexpected error while running command.
Command: rpm -q --queryformat '%{version} %{release} %{vendor}' python-cinder
Exit code: 1
Stdout: u'package python-cinder is not installed\n'
Stderr: u''.
2017-06-08 09:19:09.120 7 INFO cinder.volume.drivers.netapp.utils [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Could not run dpkg-query command: [Errno 2] No such file or directory.
2017-06-08 09:19:09.121 7 INFO cinder.volume.drivers.netapp.common [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] OpenStack OS Version Info: 10.0.0|10.0.0|unknown vendor|Linux-3.10.0-514.10.2.el7.x86_64-x86_64-with-centos-7.3.1611-Core
2017-06-08 09:19:09.122 7 INFO cinder.volume.drivers.netapp.common [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Requested unified config: ontap_cluster and iscsi.
2017-06-08 09:19:09.165 7 INFO cinder.volume.drivers.netapp.common [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] NetApp driver of family ontap_cluster and protocol iscsi loaded.
2017-06-08 09:19:09.168 7 INFO cinder.volume.manager [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Image-volume cache disabled for host compute-1.mgmt.openstack.iq.pl@NettapISCSI.
2017-06-08 09:19:09.170 7 INFO oslo_service.service [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Starting 1 workers
2017-06-08 09:19:09.176 7 WARNING oslo_config.cfg [req-6944eed3-df6b-4423-bda5-7bca3ad5a051 - - - - -] Option "enable_v1_api" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-06-08 09:19:09.177 199 INFO cinder.service [-] Starting cinder-volume node (version 10.0.0)
2017-06-08 09:19:09.194 199 INFO cinder.coordination [-] Coordination backend started successfully.
2017-06-08 09:19:09.196 199 INFO cinder.volume.manager [req-10414982-90f9-4583-a216-f915382e5b5b - - - - -] Starting volume driver NetAppCmodeISCSIDriver (1.0.0)
2017-06-08 09:19:09.303 199 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode [req-10414982-90f9-4583-a216-f915382e5b5b - - - - -] Could not get performance base counter name. Performance-based scheduler functions may not be available.
2017-06-08 09:19:09.303 199 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode Traceback (most recent call last):
2017-06-08 09:19:09.303 199 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/performance/perf_cmode.py", line 47, in _init_counter_info
2017-06-08 09:19:09.303 199 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode 'avg_processor_busy'))
2017-06-08 09:19:09.303 199 ERROR cinder.volume.drivers.netapp.dataontap.performance.perf_cmode File "/var/l...

[sklgromek (t-gromowski)]

This looks like an api problem in ontap version 9.
I tried running the cinder with virtual appliance versions 8.1.4 and 9.0.
In version 8 everything works fine, in version 9 is the problem as above.

[sklgromek (t-gromowski)]

I solve the problem in my environment.
It looks like the driver does not work at all with "Vserver Scoped" permissions.
Vserver scoped user can not get object system:constituent.
With Cluster permissions scoped user everything work.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/499148

[Chris Suttles (killface007)]

I have the same experience. NO problems on ontap 8, we upgraded to 9, and things broke. We patched around it but the proposed fix looks much cleaner. :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499148
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=887797541dff6d2cd10265de26214bcf1515fcf7
Submitter: Zuul
Branch: master

commit 887797541dff6d2cd10265de26214bcf1515fcf7
Author: Chuck Fouts <email address hidden>
Date: Mon Jul 3 16:44:41 2017 -0400

    NetApp: Fix to support SVM scoped permissions.

    Improves NetApp cDOT block and file drivers suport for SVM scoped user
    accounts. Features not supported for SVM scoped users include QoS,
    aggregate usage reporting, and dedupe usage reporting.

    Change-Id: I2b42622dbbb0f9f9f3eb9081cf67fb27e6c1a218
    Closes-Bug: #1694579

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/572382

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572382
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0cc92ee4b5d3f4c87ed40685246537c7fbfa1891
Submitter: Zuul
Branch: stable/pike

commit 0cc92ee4b5d3f4c87ed40685246537c7fbfa1891
Author: Chuck Fouts <email address hidden>
Date: Mon Jul 3 16:44:41 2017 -0400

    NetApp: Fix to support SVM scoped permissions.

    Improves NetApp cDOT block and file drivers suport for SVM scoped user
    accounts. Features not supported for SVM scoped users include QoS,
    aggregate usage reporting, and dedupe usage reporting.

    Change-Id: I2b42622dbbb0f9f9f3eb9081cf67fb27e6c1a218
    Closes-Bug: #1694579
    (cherry picked from commit 887797541dff6d2cd10265de26214bcf1515fcf7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.1.1 release.

[Purandhar Sairam Mannidi (sai438)]

Can we get a backport of this fix on driverfixes/newton branch?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/581066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581066
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d67448fdad668a35c5c35a5a06d2ac2af5b26bcd
Submitter: Zuul
Branch: stable/ocata

commit d67448fdad668a35c5c35a5a06d2ac2af5b26bcd
Author: Chuck Fouts <email address hidden>
Date: Mon Jul 3 16:44:41 2017 -0400

    NetApp: Fix to support SVM scoped permissions.

    Improves NetApp cDOT block and file drivers suport for SVM scoped user
    accounts. Features not supported for SVM scoped users include QoS,
    aggregate usage reporting, and dedupe usage reporting.

    Change-Id: I2b42622dbbb0f9f9f3eb9081cf67fb27e6c1a218
    Closes-Bug: #1694579
    (cherry picked from commit 887797541dff6d2cd10265de26214bcf1515fcf7)
    Conflicts:
            cinder/tests/unit/volume/drivers/netapp/dataontap/test_block_cmode.py
     cinder/volume/drivers/netapp/dataontap/block_cmode.py
     cinder/volume/drivers/netapp/dataontap/client/client_cmode.py
     cinder/volume/drivers/netapp/dataontap/performance/perf_cmode.py

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.8 release.