Comment 1 for bug 2035572

Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Some steps to reproduce this bug by causing a "NotFound" error.. (although the error doesn't appear to be the same in the log in the description) :

Set manila's debug logging

$ manila create nfs 1 --name testshare
$ manila access-allow testshare ip 10.10.10.10

check manila logs for the export ID of the rule that was just created; for example if it is 1001; run:

 sudo dbus-send --print-reply \
    --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr \
    org.ganesha.nfsd.exportmgr.RemoveExport uint16:1001

on the node where ganesha is running; then try to delete the share:

$ manila delete testshare

list the shares, the share must be "error_deleting"

+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| ID | Name | Size | Share Proto | Status | Is Public | Share Type Name | Host | Availability Zone |
+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+
| b0b24d51-7785-415e-9c10-86218ad727a0 | testshare | 1 | NFS | error_deleting | False | cephfsnfstype | zorilla-cephnfs-rpm@cephfsnfs1#cephfs | manila-zone-0 |
+--------------------------------------+-----------+------+-------------+----------------+-----------+-----------------+---------------------------------------+-------------------+

manila-share log shows the following:
DEBUG oslo_concurrency.lockutils [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Lock "locked_access_rules_operation_by_share_instance_97bf7384-a6ad-43ea-a2da-1b19803babf7" "released" by "manila.share.access.locked_access_rules_operation.<locals>.wrapped.<locals>.locked_operation" :: held 0.020s {{(pid=593653) inner /usr/local/lib/python3.9/site-packages/oslo_concurrency/lockutils.py:423}}
INFO manila.message.api [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Creating message record for request_id = req-cc0acbf6-22af-4810-9159-64b7fc6eda98
ERROR oslo_messaging.rpc.server [None req-cc0acbf6-22af-4810-9159-64b7fc6eda98 demo None] Exception during message handling: manila.exception.GaneshaCommandFailure: Ganesha management command failed.
Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002
Exit code: 1
Stdout: ''
Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n'
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 233, in _execute
ERROR oslo_messaging.rpc.server return execute(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/utils.py", line 59, in __call__
ERROR oslo_messaging.rpc.server return self.execute(*args, **exkwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 115, in execute
ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_concurrency/processutils.py", line 438, in execute
ERROR oslo_messaging.rpc.server raise ProcessExecutionError(exit_code=_returncode,
ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002
ERROR oslo_messaging.rpc.server Exit code: 1
ERROR oslo_messaging.rpc.server Stdout: ''
ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n'
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 236, in wrapped
ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 579, in wrapper
ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3529, in delete_share_instance
ERROR oslo_messaging.rpc.server self.message_api.create(
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
ERROR oslo_messaging.rpc.server raise self.value
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3507, in delete_share_instance
ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules(
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 299, in update_access_rules
ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id,
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 336, in _update_access_rules
ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver(
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 401, in _update_rules_through_share_driver
ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access(
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/cephfs/driver.py", line 546, in update_access
ERROR oslo_messaging.rpc.server return self.protocol_helper.update_access(
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/__init__.py", line 314, in update_access
ERROR oslo_messaging.rpc.server self.ganesha.remove_export(share['name'])
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 516, in remove_export
ERROR oslo_messaging.rpc.server self._remove_export_dbus(confdict["EXPORT"]["Export_Id"])
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 417, in _remove_export_dbus
ERROR oslo_messaging.rpc.server self._dbus_send_ganesha("RemoveExport", "uint16:%d" % xid)
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 410, in _dbus_send_ganesha
ERROR oslo_messaging.rpc.server self.execute("dbus-send", "--print-reply", "--system",
ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/manager.py", line 240, in _execute
ERROR oslo_messaging.rpc.server raise exception.GaneshaCommandFailure(
ERROR oslo_messaging.rpc.server manila.exception.GaneshaCommandFailure: Ganesha management command failed.
ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:1002
ERROR oslo_messaging.rpc.server Exit code: 1
ERROR oslo_messaging.rpc.server Stdout: ''
ERROR oslo_messaging.rpc.server Stderr: 'Error org.freedesktop.DBus.Error.InvalidArgs: lookup_export failed with Export id not found\n'
ERROR oslo_messaging.rpc.server