[SRU] Manila driver error with ONTAP SVM-scoped user

[Kim, Jinhak]

************** SRU DESCRIPTION AT THE BOTTOM *************

With the same NetApp stanza in the manila.conf file which was used without any issue in the Zed release was used in the Bobcat release. In the Bobcat release, the share creation worked normally, but adding access rule was not worked and couldn't delete the share. Below is the error log that occurs when adding a rule.
I set all the roles indicated in NetApp's OpenStack operation guide in storage side(https://netapp-openstack-dev.github.io/openstack-docs/bobcat/manila/configuration/ontap_configuration/section_ontap-config.html#ontap-prerequisites).

########### manila-share.log ############

2024-05-27 15:43:14.708 19 INFO oslo.messaging.notification.share.create.end [None req-4b46bc06-9332-40f3-9ef0-57895519228c c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4 76a637a88d624e3ea80b261a4c66dc2a - - - -] {"message_id": "d9fcc12a-5449-437c-85a0-eb5bdddab553", "publisher_id": "share.dc1-infra-rnd-stack-ctrl-01@c400", "event_type": "share.create.end", "priority": "INFO", "payload": {"share_id": "68e79de3-5e22-472b-a895-c79e0b677b01", "user_id": "c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4", "project_id": "76a637a88d624e3ea80b261a4c66dc2a", "snapshot_id": null, "share_group_id": null, "size": 20, "name": "asdasd", "description": null, "proto": "NFS", "is_public": true, "availability_zone": null, "host": "dc1-infra-rnd-stack-ctrl-01@c400#N1_Data", "status": "creating", "share_type_id": "40cdd81c-1fa8-4fc6-8f5e-288d0b9f5430", "share_type": "NFS_VOLUME"}, "timestamp": "2024-05-27 06:43:14.708153"}
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server [None req-be5d1bf0-c013-47ac-94bc-2de599a3862f c2e47ee4c8295d950db5757f73dfe9b5149947ccf5dc4e4ba3370c210217bcc4 76a637a88d624e3ea80b261a4c66dc2a - - - -] Exception during message handling: manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 15661:entry doesn't exist
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 236, in wrapped
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/utils.py", line 481, in wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 4177, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.update_access_for_instances(context, [share_instance_id],
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/manager.py", line 4191, in update_access_for_instances
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 299, in update_access_rules
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id,
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 336, in _update_access_rules
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/access.py", line 401, in _update_rules_through_share_driver
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/drv_single_svm.py", line 103, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server self.library.update_access(context, share, access_rules, add_rules,
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py", line 2355, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server helper.update_access(share, share_name, access_rules)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/base.py", line 34, in wrapped_func
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/base.py", line 32, in source_func
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/nfs_cmode.py", line 114, in update_access
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server auth_methods = self._get_auth_methods()
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/protocols/nfs_cmode.py", line 221, in _get_auth_methods
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server kerberos_enabled = self._client.is_kerberos_enabled()
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/utils.py", line 115, in trace_wrapper
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = f(self, *args, **kwargs)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/client_cmode.py", line 2042, in is_kerberos_enabled
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server result = self.send_request('kerberos-config-get', api_args)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/client_base.py", line 89, in send_request
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self.connection.invoke_successfully(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 717, in invoke_successfully
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server return self.get_client(use_zapi=use_zapi).invoke_successfully(
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib64/python3.9/site-packages/manila/share/drivers/netapp/dataontap/client/api.py", line 388, in invoke_successfully
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server raise NaApiError(code, msg)
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server manila.share.drivers.netapp.dataontap.client.api.NaApiError: NetApp API failed. Reason - 15661:entry doesn't exist
2024-05-27 15:43:57.077 19 ERROR oslo_messaging.rpc.server
2024-05-27 15:44:08.487 19 INFO manila.share.manager [None req-4bfe58a9-a794-497d-8b75-7ee098ea0e11 - - - - - -] Updating share status

===============
SRU DESCRIPTION
===============

[Impact]

The NetApp driver kerberos-config-get fails when using a SVM-scoped user because it does have not enough privileges to perform that check. This failure causes the entire stack to fail, thus preventing access rules from being added to shares. The fix addresses this by capturing the exception and not reraising it, allowing the operation to continue.

[Test case]

Testing around this is limited because:
1) The NetApp CI upstream is broken at this time. The fix was validated internally by contributors and NetApp driver maintainers.
2) We do not have a NetApp box in our lab to verify the SRU for this scenario.
3) Running the Manila tempest suite is useless because the change is limited in scope to the NetApp driver, that is only operational when using NetApp storage.

(For SRU team) please check test steps suggestion/discussion in comments 28, 36.

UPDATE: 4) it was agreed internally that unit tests would suffice for this. I wrote and performed verification with unit tests. See comment #47.

[Regression Potential]

Given that the change is limited to the NetApp driver, it is small and was peer-validated, we consider the regression potential minimal.

[Other Info]

None.

[Gireesh Awasthi (agireesh)]

kerberos-config-get ZAPI works only when you have LIF configured with Kerberos

When We are NOT hitting this issue
- When ALL LIF of vserver is configure with Kerberos, either it disabled or enabled
- When Kerberos is configured and disable for few LIF and validation first done for disabled Kerberos LIF.

When we are going to hit this issue
- When Kerberose is not configure to any LIF
- When Kerberos is not configured for one LIF but rest of the LIFs it is configured and enabled
- When Kerberos is not configured for one LIF but rest of the LIFs it is configure and disable but first validation is done for the LIF for which Kerberos is not configured.

Propose Fix
----------------
1. Need to add one try/except block to make sure we will not hit the issue for those LIF which are not configured to Kerberos.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Carlos Eduardo <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/922768
Reason: Abandoning in favor of: https://review.opendev.org/c/openstack/manila/+/922808

[Carlos da Silva (silvacarlose)]

Hello! We are hitting this issue in a couple of deployments as well, but we started seeing this in the Wallaby branch.
We also see the issue happening with a new user that was granted administrator permissions, so I don't believe the issue is restricted to user scoped accounts, it can also affect other accounts.

[Douglas Viroel (dviroel)]

I understand that this issue might be related with missing permissions on svm-scoped accounts, but does not explain on how the bug reported only faced this problem in Bobcat release, since this code was added in Wallaby. Unless the user account was also changed when upgrading from Zed -> Bobcat. Could this be also related with newer versions of ONTAP?

[Goutham Pacha Ravi (gouthamr)]

i'll extend the question; how do you use kerberos with NetApp's DHSS=False and DHSS=True modes? It'd be super useful if there's a tutorial/documentation. I was looking here: https://netapp-openstack-dev.github.io/openstack-docs/antelope/ch_manila.html but there's no mention of kerberos

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/manila/+/922894

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/manila/+/922895

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/manila/+/922896

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922808
Committed: https://opendev.org/openstack/manila/commit/232627db3b39163cfbef00d1e625fcd9a524253a
Submitter: "Zuul (22348)"
Branch: master

commit 232627db3b39163cfbef00d1e625fcd9a524253a
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/manila/+/922942

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/manila/+/922943

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/xena
Review: https://review.opendev.org/c/openstack/manila/+/922945

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/922946

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922894
Committed: https://opendev.org/openstack/manila/commit/44df1ec42ddc044592b6bdcbe685311677bbbceb
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 44df1ec42ddc044592b6bdcbe685311677bbbceb
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922895
Committed: https://opendev.org/openstack/manila/commit/78b4978992b60b9be5170753d3076b439b81bb42
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 78b4978992b60b9be5170753d3076b439b81bb42
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922896
Committed: https://opendev.org/openstack/manila/commit/0e3523e1168e632618dc3ed550869e42b3979aaa
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 0e3523e1168e632618dc3ed550869e42b3979aaa
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)
    (cherry picked from commit 78b4978992b60b9be5170753d3076b439b81bb42)

[Jonathan Williams (jon-at-netapp)]

Just a observation on this issue.

Prior to the kerberos-config-get-iter call there is a net-interface-get-iter call. The purpose of the
net-interface-get-iter call is to fetch LIFs from the SVM.

What I've observed is:

- The net-interface-get-iter call fetches all LIFs from the SVM (including management LIFs)

- Manila than grabs the first LIF from the net-interface-get-iter response?

If this LIF happens to be a management LIf (instead of a data LIF) the workflow breaks. The reason for this can be seen in the API response. For a data LIF, kerberos-config-get-iter returns a valid response
even if there is no kerberos configuration. For a management LIF, kerberos-config-get-iter simply returns a null response (because there can be no kerberos configuration associated with a management LIF):

data lif:

<?xml version='1.0' encoding='UTF-8' ?>
<netapp version='1.221' xmlns='http://www.netapp.com/filer/admin'>

    <!-- Output of kerberos-config-get-iter [Execution Time: 56 ms] -->
     <results status='passed'>
          <attributes-list>
               <kerberos-config-info>
                    <interface-name>manila-nfs-data-01</interface-name>
                    <is-kerberos-enabled>false</is-kerberos-enabled>
                    <vserver>demo-manila-svm</vserver>
               </kerberos-config-info>
          </attributes-list>
          <num-records>1</num-records>
     </results>
</netapp>

mgmt lif:

<?xml version='1.0' encoding='UTF-8' ?>
<netapp version='1.221' xmlns='http://www.netapp.com/filer/admin'>

    <!-- Output of kerberos-config-get-iter [Execution Time: 48 ms] -->
     <results status='passed'>
          <num-records>0</num-records>
     </results>
</netapp>

I might be misreading the fix, but it seems to just add logic to handle the API exception. Perhaps
a more elegant solution would be for Manila not to randomly select a SVM management LIF in the first place. And to ensure that kerberos-config-get-iter is only ever run against a data LIF (perhaps by adding a ‘default-data-files’ service policy parameter to the net-interface-get-iter call).

My fear is that you'll continue to have issues in that:

- Manila runs kerberos-config-get-iter against a SVM management LIF
- Manila thinks that there is no kerberos configuration...hence it doesn't need to worry about Kerberos
auth
- There actually is a Kerberos configuration present (albeit on the data LIFs)

[Brian Murray (brian-murray)]

Ubuntu 23.10 (Mantic Minotaur) has reached end of life, so this bug will not be fixed for that specific release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922943
Committed: https://opendev.org/openstack/manila/commit/2aba4ca70e7f4b01c7b65c62b91b603a3ef9ddbe
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 2aba4ca70e7f4b01c7b65c62b91b603a3ef9ddbe
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)
    (cherry picked from commit 78b4978992b60b9be5170753d3076b439b81bb42)
    (cherry picked from commit 0e3523e1168e632618dc3ed550869e42b3979aaa)
    (cherry picked from commit 356071ba7060487a2d6658ab75c9b7b4d1d0149f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/922945
Committed: https://opendev.org/openstack/manila/commit/b2b152d93c7066602d7d3e9da591d3b88434ea67
Submitter: "Zuul (22348)"
Branch: unmaintained/xena

commit b2b152d93c7066602d7d3e9da591d3b88434ea67
Author: agireesh <email address hidden>
Date: Wed Jun 26 17:36:01 2024 +0530

    Fixed bug 2069125: Manila driver error with ONTAP SVM-scoped user

    Fixed the issue for the NetApp driver in the ZAPI workflow where a
    Vserver-scoped user failed to add rules for a share when the
    Vserver LIF was not configured with Kerberos.

    Closes-Bug: #2069125
    Change-Id: I8544efbdac207c54d75c55ee4a039c5324433794
    (cherry picked from commit 232627db3b39163cfbef00d1e625fcd9a524253a)
    (cherry picked from commit 44df1ec42ddc044592b6bdcbe685311677bbbceb)
    (cherry picked from commit 78b4978992b60b9be5170753d3076b439b81bb42)
    (cherry picked from commit 0e3523e1168e632618dc3ed550869e42b3979aaa)
    (cherry picked from commit 356071ba7060487a2d6658ab75c9b7b4d1d0149f)
    (cherry picked from commit 2aba4ca70e7f4b01c7b65c62b91b603a3ef9ddbe)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 16.2.1 release.

[Mauricio Faria de Oliveira (mfo)]

The fix is present in Ubuntu Oracular (development series).

In the future, please update the bug task's status to Fix Released in such cases (and clarify in Other Info if needed). Thanks! :)

$ pull-lp-source manila oracular
Found manila 1:18.1.0+git2024080716.14ef8916-0ubuntu2 in oracular
...

$ grep -F -B1 -A7 \
  '# Catch the exception in case kerberos is not configured with LIF.' \
  manila-18.1.0+git2024080716.14ef8916/manila/share/drivers/netapp/dataontap/client/client_cmode.py
            result = None
            # Catch the exception in case kerberos is not configured with LIF.
            try:
                result = self.send_request('kerberos-config-get', api_args)
            except netapp_api.NaApiError as e:
                with excutils.save_and_reraise_exception() as exc_context:
                    if "entry doesn't exist" in e.message:
                        exc_context.reraise = False
                        return False

[Mauricio Faria de Oliveira (mfo)]

Hi Rodrigo,

Apologies for the delay in getting to review this.

Thanks for your work on this and debdiffs.

Changes required. Happy to help with any questions.

Thanks,
Mauricio

Noble:
---

d/changelog:

```
+manila (1:18.0.1-0ubuntu3) noble; urgency=medium
 manila (1:18.0.1-0ubuntu2) noble; urgency=medium
```

The version increment in stable is .1 (not 1) for most cases.
Please read [1] for future SRUs. Happy to help with questions!

```
+ * d/p/lp2069125.patch: Fix NetApp Driver error adding
+ access rules when using SVM-scoped user and the Vserver
+ LIF is not configured with Kerberos.
```

The LP bug closing tag `(LP: #<number)` is required for SRUs.
Please see SRU requirements [2].

d/p/lp2069125.patch:

Please add DEP-3 headers [3] to the patch file.
For git patches (e.g., git format-patch or git web exported),
you usually only add two additional headers:
```
Bug-Ubuntu: https://bugs.launchpad.net/bugs/<number>
Origin: [<upstream|backport|vendor>,] https://<gitweb>/commit/<id>
```

d/p/series:

Correct.

Jammy:
---

Likewise.

Links:
---

[1] https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation#Update_the_packaging
[2] https://canonical-sru-docs.readthedocs-hosted.com/en/latest/reference/requirements/#bug-references-in-changelogs
[3] https://dep-team.pages.debian.net/deps/dep3/

[Mauricio Faria de Oliveira (mfo)]

Rodrigo,

Thanks for the clear and succint SRU template.

For `Test Steps`, I can appreciate the limitations.
However, we'd need at least a minimal way to verify the code change, even if that is synthetically or with a mocked up call (AFAICT/IIUIC the code change does look correct and safe).

I looked up the ONTAP simulator, but apparently it's only available for NetApp customers/partners (not guests who create accunts on their support site), and it requires Windows/VMware (doable, but not fast).

Feedback from the original reporter (with hardware access) to at least the version they require this on (Jammy?) would be ideal, if at all possible, and this should help build confidence on the newer version, which possibly only receives synthetic/software testing.

Thanks again!
Mauricio

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Hi Mauricio. I uploaded the new debdiffs with the changes you requested. If I am not mistaken, Mantic/Bobcat is EOL so one less debdiff.

About the test steps, I mentioned in item (1) that numerous contributors and driver maintainers have validated the fix internally and confirmed that it worked without issues. Unfortunately that's the best we have.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Mauricio, as per Ed's internal email about extending bobcat's support for 3 extra months, I have prepared and attached above the bobcat debdiff.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 19.0.0.0rc1 release candidate.

[Mauricio Faria de Oliveira (mfo)]

Hey Rodrigo o/

Thanks for the adjusted debdiffs, I'll take a look. However, for now:

Comment #32:
> About the test steps, I mentioned in item (1) that numerous contributors
> and driver maintainers have validated the fix internally and confirmed
> that it worked without issues. Unfortunately that's the best we have.

Yes, I was aware about item 1 in [Test Plan] when I wrote comment #28,
but I didn't think that was the end of options thus suggested an idea:

Comment #28:
> For `Test Steps`, I can appreciate the limitations.
> However, we'd need at least a minimal way to verify the code change,
> even if that is synthetically or with a mocked up call
> (AFAICT/IIUIC the code change does look correct and safe).
"""

I also appreciate the existing validation by contributors/drive maintainers
is 'the best we have', but is that really 'the best we _could_ have'?

Apparently, we _could_ do something.

Is it _not possible_ to do at least _a mocked up test_ for this code path?
That seems to be standard practice in test suites of openstack components.

From what I can look at the patch, one could mock `send_request` to return
normally or trigger the expected exception (with/without the right message),
and check that the caller returns or reraises as expected in the 3 cases.

Right? If that is not possible, please clarify the reason, as I understand
this SRU is needed, so we'd need to figure tests out so it can be released.

It's difficult to SRU a change that can't be _minimally_ tested in _any way_,
or even regression tested in indirect ways (to at least check it does not
break other things -- that is a valid test too, when direct tests are hard).

I'll continue to review and hopefully upload the changes, and it might even
be accepted into -proposed depending on the reviewer/discussion, but without
some way to test this, I'd imagine it would be gated at release to -updates.

Thanks again for all your work on this.

[Mauricio Faria de Oliveira (mfo)]

Hi again Rodrigo,

Thanks for the improvements in the debdiffs.
I have a few remarks that seem minor, but would help with your future uploads.

Noble:

1) The DEP-3 headers were introduced in the middle of commit message,
which alters it (in this case, between Closes-Bug/Change-Id footers).

While this might seem minor as it's not a code level change, this is
effectively altering the original patch at the metadata level.
(I recall reading an upstream project's guidelines for stable backports
that suggested a lower level of trust in backports with a commit message
that was modified from the original.
This is really not the case here, but it's good to keep best practices.)

Ideally, please use it before the commit message if you like it that way
(it's more visible/easier to review, IMHO), or after the commit message
(some people prefer it that way).

All these three ways are a style choice, not a requirement, but hopefully
the suggestions above may be helpful to you in the future/simpler reviews.

2) > Origin: upstream, https://review.opendev.org/922894

`Origin:` should use commit URL, not review/pull-request, if possible; see DEP-3.

In this case, `Origin:` links a PR with patch _merged_ in branch 'stable/2024.1'
(as seen in 'Repo | Branch' with 'stable/2024.1'), and the commit ID is found
in comment #14 (which does match/link to the PR in Origin). So, let's use this.

> Reviewed: https://review.opendev.org/c/openstack/manila/+/922894
> Committed: https://opendev.org/openstack/manila/commit/44df1ec42ddc044592b6bdcbe685311677bbbceb
> Submitter: "Zuul (22348)"
> Branch: stable/2024.1

I checked, and it applies cleanly (thus the `upstream` keyword is correct).

$ curl -fsSL 'https://opendev.org/openstack/manila/commit/44df1ec42ddc044592b6bdcbe685311677bbbceb.patch' >/tmp/test.patch
$ patch -p1 -F0 --dry-run -i /tmp/test.patch
checking file manila/share/drivers/netapp/dataontap/client/client_cmode.py
checking file releasenotes/notes/bug-2069125-fix-manila-driver-error-with-ontap-svm-scoped-user-when-add-rule-1ae120a96dd8f68a.yaml

Jammy (Yoga):

Patch is reversed (could apply with `patch -R`).
Apparently the order of `.dsc` files was swapped.

Likewise Noble on DEP-3 and Origin:

comment #23:

> Reviewed: https://review.opendev.org/c/openstack/manila/+/922943
> Committed: https://opendev.org/openstack/manila/commit/2aba4ca70e7f4b01c7b65c62b91b603a3ef9ddbe
> Submitter: "Zuul (22348)"
> Branch: unmaintained/yoga

This time I adjusted all of that, since it's mostly style,
but I suggest you could consider these suggestions as good
practices for SRUs, and you're already achieving 90% of it
anyway :) so a small improvement should give you no issues
with any reviewer (if there's anyone more pedantic. :)

Uploaded to Noble and Jammy; checking Cloud Archive tomorrow.

Thanks again!
Mauricio

[Brian Murray (brian-murray)]

Hello Kim, or anyone else affected,

Accepted manila into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/manila/1:18.0.1-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

While I have accepted this into -proposed for Noble and Jammy I've added blocked-proposed tags so that the test case can be improved before the package is released to -updates.

[Brian Murray (brian-murray)]

Hello Kim, or anyone else affected,

Accepted manila into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/manila/1:14.1.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/931985

[James Page (james-page)]

Hello Kim, or anyone else affected,

Accepted manila into caracal-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:caracal-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-caracal-needed to verification-caracal-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-caracal-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Kim, or anyone else affected,

Accepted manila into yoga-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:yoga-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-yoga-needed to verification-yoga-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-yoga-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/931985
Committed: https://opendev.org/openstack/manila/commit/31c97211cbfab7d6fcc96ba92fbbc9aa988c167b
Submitter: "Zuul (22348)"
Branch: master

commit 31c97211cbfab7d6fcc96ba92fbbc9aa988c167b
Author: Rodrigo Barbieri <email address hidden>
Date: Wed Oct 9 16:09:56 2024 -0300

    Improve unit tests in NetApp Driver

    Change I8544efbdac207c54d75c55ee4a039c5324433794
    was missing unit tests.

    Related-bug: #2069125
    Change-Id: I9ab37f9d225a84fcceb9b618fad7d110974d2ed4

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Performed verification with unit tests written to the test the code changes. The unit tests were based on [1] which is already merged upstream on master, but had to be adjusted to work on caracal and yoga because of lack of [2] in those branches.

In noble, the tox.ini file is incorrectly set to install libraries according to master constraints, therefore the unit tests do not work unless tox.ini is fixed. I copied the tox.in from stable/2024.1 from manila upstream repo.

[1] https://review.opendev.org/c/openstack/manila/+/931985
[2] https://review.opendev.org/c/openstack/manila/+/926903

[Rodrigo Barbieri (rodrigo-barbieri2010)]

@brian-murray: Verification was performed according to the updated test case which was based on internally agreement that unit would suffice. Could you please re-review and remove the block-proposed tags if everything is all right now? Thanks in advance

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 18.2.0 release.

[Andreas Hasenack (ahasenack)]

I checked the noble caracal verification and spotted the two new tests, and they passed:

{0} manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_raise [0.023855s] ... ok
{2} manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_return_false [0.027121s] ... ok
{2} manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_raise [0.186970s] ... ok
{0} manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_return_false [0.198930s] ... ok
manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_return_false 0.199
manila.tests.share.drivers.netapp.dataontap.client.test_client_cmode.NetAppClientCmodeTestCase.test_is_kerberos_enabled_exception_raise 0.187

And the test was done with manila from noble-proposed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package manila - 1:18.0.1-0ubuntu2.1

---------------
manila (1:18.0.1-0ubuntu2.1) noble; urgency=medium

  * d/p/lp2069125.patch: Fix NetApp Driver error adding
    access rules when using SVM-scoped user and the Vserver
    LIF is not configured with Kerberos. (LP: #2069125)

 -- Rodrigo Barbieri <email address hidden> Tue, 03 Sep 2024 14:15:13 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for manila has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andreas Hasenack (ahasenack)]

Same for jammy-yoga (jammy caracal is 18.0.1 (from noble) on jammy, so not the normal ubuntu sru, but cloud-archive I guess).

[Launchpad Janitor (janitor)]

This bug was fixed in the package manila - 1:14.1.1-0ubuntu1.1

---------------
manila (1:14.1.1-0ubuntu1.1) jammy; urgency=medium

  * d/p/lp2069125.patch: Fix NetApp Driver error adding
    access rules when using SVM-scoped user and the Vserver
    LIF is not configured with Kerberos. (LP: #2069125)

 -- Rodrigo Barbieri <email address hidden> Tue, 03 Sep 2024 19:38:11 +0000

[Edward Hope-Morley (hopem)]

The verification of the Stable Release Update for manila has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Edward Hope-Morley (hopem)]

This bug was fixed in the package manila - 1:14.1.1-0ubuntu1.1~cloud0
---------------

 manila (1:14.1.1-0ubuntu1.1~cloud0) focal-yoga; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 manila (1:14.1.1-0ubuntu1.1) jammy; urgency=medium
 .
   * d/p/lp2069125.patch: Fix NetApp Driver error adding
     access rules when using SVM-scoped user and the Vserver
     LIF is not configured with Kerberos. (LP: #2069125)

[Rodrigo Barbieri (rodrigo-barbieri2010)]

I just double checked the code on all SRUs marked as Fix Released, and found out that the Caracal UCA was marked as fix-released but it was not in the code. Comment #42 marked it fix-released by mistake I suppose. It is NOT YET released.

The fix is still sitting in the proposed pocket:

 manila | 1:18.0.1-0ubuntu2~cloud0 | caracal | jammy-updates | source
 manila | 1:18.0.1-0ubuntu2.1~cloud0 | caracal-proposed | jammy-proposed | source

[Edward Hope-Morley (hopem)]

The verification of the Stable Release Update for manila has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Edward Hope-Morley (hopem)]

This bug was fixed in the package manila - 1:18.0.1-0ubuntu2.1~cloud0
---------------

 manila (1:18.0.1-0ubuntu2.1~cloud0) jammy-caracal; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 manila (1:18.0.1-0ubuntu2.1) noble; urgency=medium
 .
   * d/p/lp2069125.patch: Fix NetApp Driver error adding
     access rules when using SVM-scoped user and the Vserver
     LIF is not configured with Kerberos. (LP: #2069125)

[Mauricio Faria de Oliveira (mfo)]

Uploaded to UCA Bobcat.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 17.3.0 release.

[James Page (james-page)]

Hello Kim, or anyone else affected,

Accepted manila into bobcat-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:bobcat-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-bobcat-needed to verification-bobcat-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-bobcat-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Verification on bobcat done, see output above.

[Mauricio Faria de Oliveira (mfo)]

Uploaded to UCA Antelope.

[Edward Hope-Morley (hopem)]

Hello Kim, or anyone else affected,

Accepted manila into antelope-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:antelope-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-antelope-needed to verification-antelope-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-antelope-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/manila/+/935181

[Rodrigo Barbieri (rodrigo-barbieri2010)]

Verification done, see attachment above

[Edward Hope-Morley (hopem)]

The verification of the Stable Release Update for manila has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Edward Hope-Morley (hopem)]

This bug was fixed in the package manila - 1:17.1.0-0ubuntu1~cloud1
---------------

 manila (1:17.1.0-0ubuntu1~cloud1) jammy-bobcat; urgency=medium
 .
   * d/p/lp2069125.patch: Fix NetApp Driver error adding
     access rules when using SVM-scoped user and the Vserver
     LIF is not configured with Kerberos. (LP: #2069125)

[Edward Hope-Morley (hopem)]

The verification of the Stable Release Update for manila has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Edward Hope-Morley (hopem)]

This bug was fixed in the package manila - 1:16.1.0-0ubuntu1~cloud1
---------------

 manila (1:16.1.0-0ubuntu1~cloud1) jammy-antelope; urgency=medium
 .
   * d/p/lp2069125.patch: Fix NetApp Driver error adding
     access rules when using SVM-scoped user and the Vserver
     LIF is not configured with Kerberos. (LP: #2069125)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dr. Jens Harbott <email address hidden>" on branch: unmaintained/wallaby
Review: https://review.opendev.org/c/openstack/manila/+/922946
Reason: unmaintained/wallaby branch of openstack/manila transitioned to End of Life and is about to be deleted. To be able to do that, all open patches need to be abandoned.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila xena-eol Xena release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila yoga-eol Yoga release.