[SRU] Manila driver error with ONTAP SVM-scoped user

[Kim, Jinhak]

************** SRU DESCRIPTION AT THE BOTTOM *************

With the same NetApp stanza in the manila.conf file which was used without any issue in the Zed release was used in the Bobcat release. In the Bobcat release, the share creation worked normally, but adding access rule was not worked and couldn't delete the share. Below is the error log that occurs when adding a rule.
I set all the roles indicated in NetApp's OpenStack operation guide in storage side(https://netapp-openstack-dev.github.io/openstack-docs/bobcat/manila/configuration/ontap_configuration/section_ontap-config.html#ontap-prerequisites).

########### manila-share.log ############

2024-05-27 15:43:14.708 19 INFO oslo.messaging.notification.share.create.end [None req-4b46bc06-9332-40f3-9ef0-57895519228c c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4 76a637a88d624e3ea80b261a4c66dc2a - - - -] {"message_id": "d9fcc12a-5449-437c-85a0-eb5bdddab553", "publisher_id": "share.dc1-infra-rnd-stack-ctrl-01@c400", "event_type": "share.create.end", "priority": "INFO", "payload": {"share_id": "68e79de3-5e22-472b-a895-c79e0b677b01", "user_id": "c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4", "project_id": "76a637a88d624e3ea80b261a4c66dc2a", "snapshot_id": null, "share_group_id": null, "size": 20, "name": "asdasd", "description": null, "proto": "NFS", "is_public": true, "availability_zone": null, "host": "dc1-infra-rnd-stack-ctrl-01@c400#N1_Data", "status": "creating", "share_type_id": "40cdd81c-1fa8-4fc6-8f5e-288d0b9f5430", "share_type": "NFS_VOLUME"}, "timestamp": "2024-05-27 06:43:14.708153"}
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server [None req-be5d1bf0-c013-47ac-94bc-2de599a3862f c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4 76a637a88d624e3ea80b261a4c66dc2a - - - -] Exception during message handling: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 15661:entry doesn't exist
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 236, in wrapped
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/utils.py", line 481, in wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 4177, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.update_access_for_instances(context, [share_instance_id],
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 4191, in update_access_for_instances
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 299, in update_access_rules
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id,
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 336, in _update_access_rules
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 401, in _update_rules_through_share_driver
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/drv_single_svm.py", line 103, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.library.update_access(context, share, access_rules, add_rules,
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 2355, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server helper.update_access(share, share_name, access_rules)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/base.py", line 34, in wrapped_func
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/base.py", line 32, in source_func
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/nfs_cmode.py", line 114, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server auth_methods = self._get_auth_methods()
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/nfs_cmode.py", line 221, in _get_auth_methods
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server kerberos_enabled = self._client.is_kerberos_enabled()
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 2042, in is_kerberos_enabled
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = self.send_request('kerberos-config-get', api_args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/client_base.py", line 89, in send_request
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self.connection.invoke_successfully(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 717, in invoke_successfully
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self.get_client(use_zapi=use_zapi).invoke_successfully(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 388, in invoke_successfully
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server raise NaApiError(code, msg)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 15661:entry doesn't exist
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server
2024-05-27 15:44:08.487 19 INFO manila.share.manager [None req-4bfe58a9-a794-497d-8b75-7ee098ea0e11 - - - - - -] Updating share status

===============
SRU DESCRIPTION
===============

[Impact]

The NetApp driver kerberos-config-get fails when using a SVM-scoped user because it does have not enough privileges to perform that check. This failure causes the entire stack to fail, thus preventing access rules from being added to shares. The fix addresses this by capturing the exception and not reraising it, allowing the operation to continue.

[Test case]

Testing around this is limited because:
1) The NetApp CI upstream is broken at this time. The fix was validated internally by contributors and NetApp driver maintainers.
2) We do not have a NetApp box in our lab to verify the SRU for this scenario.
3) Running the Manila tempest suite is useless because the change is limited in scope to the NetApp driver, that is only operational when using NetApp storage.

[Regression Potential]

Given that the change is limited to the NetApp driver, it is small and was peer-validated, we consider the regression potential minimal.

[Other Info]

None.

[Gireesh Awasthi (agireesh)]

kerberos-config-get ZAPI works only when you have LIF configured with Kerberos

When We are NOT hitting this issue
- When ALL LIF of vserver is configure with Kerberos, either it disabled or enabled
- When Kerberos is configured and disable for few LIF and validation first done for disabled Kerberos LIF.

When we are going to hit this issue
- When Kerberose is not configure to any LIF
- When Kerberos is not configured for one LIF but rest of the LIFs it is configured and enabled
- When Kerberos is not configured for one LIF but rest of the LIFs it is configure and disable but first validation is done for the LIF for which Kerberos is not configured.

Propose Fix
----------------
1. Need to add one try/except block to make sure we will not hit the issue for those LIF which are not configured to Kerberos.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Carlos Eduardo <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/922768
Reason: Abandoning in favor of: https://review.opendev.org/c/openstack/manila/+/922808

[Carlos Eduardo (silvacarlose)]

Hello! We are hitting this issue in a couple of deployments as well, but we started seeing this in the Wallaby branch.
We also see the issue happening with a new user that was granted administrator permissions, so I don't believe the issue is restricted to user scoped accounts, it can also affect other accounts.

[Douglas Viroel (dviroel)]

I understand that this issue might be related with missing permissions on svm-scoped accounts, but does not explain on how the bug reported only faced this problem in Bobcat release, since this code was added in Wallaby. Unless the user account was also changed when upgrading from Zed -> Bobcat. Could this be also related with newer versions of ONTAP?

[Goutham Pacha Ravi (gouthamr)]

i'll extend the question; how do you use kerberos with NetApp's DHSS=False and DHSS=True modes? It'd be super useful if there's a tutorial/documentation. I was looking here: https://netapp-openstack-dev.github.io/openstack-docs/antelope/ch_manila.html but there's no mention of kerberos

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/manila/+/922894

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/manila/+/922895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/manila/+/922896

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922808
Committed: https://opendev.org/openstack/manila/commit/232627db3b39163cfbef00d1e625fcd9a524253a
Submitter: "Zuul (22348)"
Branch: master

commit 232627db3b39163cfbef00d1e625fcd9a524253a
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/manila/+/922942

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/manila/+/922943

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/xena
Review: https://review.opendev.org/c/openstack/manila/+/922945

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/922946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922894
Committed: https://opendev.org/openstack/manila/commit/44df1ec42ddc044592b6bdcbe685311677bbbceb
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 44df1ec42ddc044592b6bdcbe685311677bbbceb
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922895
Committed: https://opendev.org/openstack/manila/commit/78b4978992b60b9be5170753d3076b439b81bb42
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 78b4978992b60b9be5170753d3076b439b81bb42
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922896
Committed: https://opendev.org/openstack/manila/commit/0e3523e1168e632618dc3ed550869e42b3979aaa
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 0e3523e1168e632618dc3ed550869e42b3979aaa
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)
    (cherry picked from commit 78b4978992b60b9be5170753d3076b439b81bb42)

[Jonathan Williams (jon-at-netapp)]

Just a observation on this issue.

Prior to the kerberos-config-get-iter call there is a net-interface-get-iter call. The purpose of the
net-interface-get-iter call is to fetch LIFs from the SVM.

What I've observed is:

- The net-interface-get-iter call fetches all LIFs from the SVM (including management LIFs)

- Manila than grabs the first LIF from the net-interface-get-iter response?

If this LIF happens to be a management LIf (instead of a data LIF) the workflow breaks. The reason for this can be seen in the API response. For a data LIF, kerberos-config-get-iter returns a valid response
even if there is no kerberos configuration. For a management LIF, kerberos-config-get-iter simply returns a null response (because there can be no kerberos configuration associated with a management LIF):

data lif:

<?xml version='1.0' encoding='UTF-8' ?>
<netapp version='1.221' xmlns='http://www.netapp.com/filer/admin'>

    <!-- Output of kerberos-config-get-iter [Execution Time: 56 ms] -->
     <results status='passed'>
          <attributes-list>
               <kerberos-config-info>
                    <interface-name>manila-nfs-data-01</interface-name>
                    <is-kerberos-enabled>false</is-kerberos-enabled>
                    <vserver>demo-manila-svm</vserver>
               </kerberos-config-info>
          </attributes-list>
          <num-records>1</num-records>
     </results>
</netapp>

mgmt lif:

<?xml version='1.0' encoding='UTF-8' ?>
<netapp version='1.221' xmlns='http://www.netapp.com/filer/admin'>

    <!-- Output of kerberos-config-get-iter [Execution Time: 48 ms] -->
     <results status='passed'>
          <num-records>0</num-records>
     </results>
</netapp>

I might be misreading the fix, but it seems to just add logic to handle the API exception. Perhaps
a more elegant solution would be for Manila not to randomly select a SVM management LIF in the first place. And to ensure that kerberos-config-get-iter is only ever run against a data LIF (perhaps by adding a ‘default-data-files’ service policy parameter to the net-interface-get-iter call).

My fear is that you'll continue to have issues in that:

- Manila runs kerberos-config-get-iter against a SVM management LIF
- Manila thinks that there is no kerberos configuration...hence it doesn't need to worry about Kerberos
auth
- There actually is a Kerberos configuration present (albeit on the data LIFs)