Force delete does not work if share service is down

[Goutham Pacha Ravi]

The ability to force delete a share [1][2] was provided as an admin API [3] to remove shares from manila despite backend failures. However, this API does not account for times when the backend share service is not responding. For example:

$ manila service-list --binary manila-share
+----+--------------+------------------+---------+---------+-------+----------------------------+
| Id | Binary | Host | Zone | Status | State | Updated_at |
+----+--------------+------------------+---------+---------+-------+----------------------------+
| 8 | manila-share | ostk-train@beta | chicago | enabled | down | 2020-03-11T17:15:58.000000 |
| 9 | manila-share | ostk-train@delta | chicago | enabled | down | 2020-03-11T17:21:15.000000 |
| 10 | manila-share | ostk-train@gamma | dallas | enabled | down | 2020-03-11T17:15:49.000000 |
| 11 | manila-share | ostk-train@alpha | dallas | enabled | down | 2020-03-11T17:15:54.000000 |
+----+--------------+------------------+---------+---------+-------+----------------------------+

In this situation, if force deleting a share on a backend reporting as "down" leaves the share in "deleting" state forever.

The error in the API logs suggests that the host binary was not found. This error isn't being handled:

ERROR manila.api.middleware.fault [None req-c3c161dc-ddc3-478f-a538-6f36ecd98bb6 None None]
Caught error: Could not find binary ostk-train@delta on host manila-share.: manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host man
ila-share.
ERROR manila.api.middleware.fault Traceback (most recent call last):
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/middleware/fault.py",
 line 77, in __call__
ERROR manila.api.middleware.fault return req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/oslo_middle
ware/base.py", line 130, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self.application)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/keystonemid
dleware/auth_token/__init__.py", line 341, in __call__
ERROR manila.api.middleware.fault response = req.get_response(self._app)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1314, in send
ERROR manila.api.middleware.fault application, catch_exc_info=False)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/reque
st.py", line 1278, in call_application
ERROR manila.api.middleware.fault app_iter = application(self.environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/routes/midd
leware.py", line 141, in __call__
ERROR manila.api.middleware.fault response = self.app(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 143, in __call__
ERROR manila.api.middleware.fault return resp(environ, start_response)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 129, in __call__
ERROR manila.api.middleware.fault resp = self.call_func(req, *args, **kw)
ERROR manila.api.middleware.fault File "/usr/local/lib/python3.6/dist-packages/webob/dec.p
y", line 193, in call_func
ERROR manila.api.middleware.fault return self.func(req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 766, in __call__
ERROR manila.api.middleware.fault content_type, body, accept)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", l
ine 830, in _process_stack
ERROR manila.api.middleware.fault action_result = self.dispatch(meth, request, action_ar
gs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 925, in dispatch
ERROR manila.api.middleware.fault return method(req=request, **action_args)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1068, in version_select
ERROR manila.api.middleware.fault return func.func(self, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 125, in update
ERROR manila.api.middleware.fault return self._update(req, id, body)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/openstack/wsgi.py", line 1162, in wrapper
ERROR manila.api.middleware.fault return f(self, req, *args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/api/v2/services.py", line 89, in _update
ERROR manila.api.middleware.fault svc = db.service_get_by_args(context, data['host'], data['binary'])
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/api.py", line 119, in service_get_by_args
ERROR manila.api.middleware.fault return IMPL.service_get_by_args(context, host, binary)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 152, in wrapper
ERROR manila.api.middleware.fault return f(*args, **kwargs)
ERROR manila.api.middleware.fault File "/opt/stack/manila/manila/db/sqlalchemy/api.py", line 462, in service_get_by_args
ERROR manila.api.middleware.fault raise exception.HostBinaryNotFound(host=host, binary=binary)
ERROR manila.api.middleware.fault manila.exception.HostBinaryNotFound: Could not find binary ostk-train@delta on host manila-share.
ERROR manila.api.middleware.fault

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-03-12-15.01.log.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/713745

[Goutham Pacha Ravi (gouthamr)]

https://review.opendev.org/713745 is not meant to be a solution, at least not yet. I put it up to start the discussion on one way to resolve this issue.

Manila's share service going down is an anomaly. We shouldn't work around such an anomaly in my opinion. Differing opinions?

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2021/manila.2021-02-11-15.00.log.html

'a "manila-manage share delete" would be a better solution, basically a tool to clean up a zombie share without ever hitting the backend'

[Vida Haririan (vhariria)]

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-04-13-15.00.log.html

[Joana Santos (josantos)]

can i be assigned to this bug?

[Wadiya Gillani (wadiya-gillani)]

Can I be assigned to this bug

[George Nyanjui (heuze)]

Please assign to this bug

[George Nyanjui (heuze)]

Please assign this bug to me

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/911694

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/911694
Committed: https://opendev.org/openstack/manila/commit/23de7b4b0cfb7c3ac4663f395298d19fa7a8776f
Submitter: "Zuul (22348)"
Branch: master

commit 23de7b4b0cfb7c3ac4663f395298d19fa7a8776f
Author: Okeke Christian <email address hidden>
Date: Thu Mar 7 12:15:20 2024 +0100

    Add delete CLI helper to ShareCommands

    This patch adds a delete cli command to ShareCommands
    The helper CLI command checks if service if up. If service
    is down, it deletes the share instance

    Closes-Bug: #1867030
    Change-Id: I6a0575c1ed86213010e50fe1b7a733cdf7fa1736