Description:
Encountered this bug in zuul logs (https://1c2b17d423b7551cb59e-a67dde8517ec7a3f9a5afba8c0473bc9.ssl.cf2.rackcdn.com/806647/6/check/manila-tempest-plugin-glusterfs-nfs/128a8a5/controller/logs/screen-m-shr.txt)
Involves line 324 in /opt/stack/manila/manila/share/drivers/glusterfs/common.py:
self.gluster_call(
'volume', args[0], self.volume, *args[1], error_policy=policy)
where the test starts multiple transactions with gluster.
Method should include a "wait for status" or "retry" command.
Log:
ERROR oslo_messaging.rpc.server [None req-cb37090d-8a6b-4771-b24d-9c43f81a000f tempest-ShareIpRulesForNFSTest-1291737832 None] Exception during message handling: manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-glusterfs-vol nfs.export-dir /share-ebf7470d-8635-4f97-9f4f-5621f5cade7d(203.0.113.6),/share-fc1428c2-7067-41cb-85d0-5f79dc99b712(1.2.3.4)' failed with details as follows:
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: Unexpected error while running command.
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: Command: sudo manila-rootwrap /etc/manila/rootwrap.conf gluster volume set manila-glusterfs-vol nfs.export-dir /share-ebf7470d-8635-4f97-9f4f-5621f5cade7d(203.0.113.6),/share-fc1428c2-7067-41cb-85d0-5f79dc99b712(1.2.3.4)
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: Exit code: 1
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: Stdout: ''
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: Stderr: 'volume set: failed: Another transaction is in progress for manila-glusterfs-vol. Please try again after some time.\n'.
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/common.py", line 196, in _gluster_call
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return gluster_execf(*(('gluster',) + args), **kwargs)
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/ganesha/utils.py", line 59, in __call__
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return self.execute(*args, **exkwargs)
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 102, in execute
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs)
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server raise ProcessExecutionError(exit_code=_returncode,
Sep 15 21:56:59.975868 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf gluster volume set manila-glusterfs-vol nfs.export-dir /share-ebf7470d-8635-4f97-9f4f-5621f5cade7d(203.0.113.6),/share-fc1428c2-7067-41cb-85d0-5f79dc99b712(1.2.3.4)
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Exit code: 1
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Stdout: ''
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Another transaction is in progress for manila-glusterfs-vol. Please try again after some time.\n'
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 216, in wrapped
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 573, in wrapper
Sep 15 21:56:59.977448 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3935, in update_access
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self.update_access_for_instances(context, [share_instance_id],
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 3949, in update_access_for_instances
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self.access_helper.update_access_rules(
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 301, in update_access_rules
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self._update_access_rules(context, share_instance_id,
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 338, in _update_access_rules
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server driver_rule_updates = self._update_rules_through_share_driver(
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/access.py", line 403, in _update_rules_through_share_driver
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server driver_rule_updates = self.driver.update_access(
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/layout.py", line 147, in update_access
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self._update_access_via_manager(gluster_mgr, context, share,
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/__init__.py", line 157, in _update_access_via_manager
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self._get_helper(gluster_mgr).update_access(
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/__init__.py", line 228, in update_access
Sep 15 21:56:59.979794 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self.gluster_manager.set_vol_option(NFS_EXPORT_DIR, export_dirs_new)
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/common.py", line 55, in wrapper
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/common.py", line 324, in set_vol_option
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server self.gluster_call(
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/glusterfs/common.py", line 208, in _gluster_call
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server raise exception.GlusterfsException(
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server manila.exception.GlusterfsException: GlusterFS management command 'volume set manila-glusterfs-vol nfs.export-dir /share-ebf7470d-8635-4f97-9f4f-5621f5cade7d(203.0.113.6),/share-fc1428c2-7067-41cb-85d0-5f79dc99b712(1.2.3.4)' failed with details as follows:
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Unexpected error while running command.
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Command: sudo manila-rootwrap /etc/manila/rootwrap.conf gluster volume set manila-glusterfs-vol nfs.export-dir /share-ebf7470d-8635-4f97-9f4f-5621f5cade7d(203.0.113.6),/share-fc1428c2-7067-41cb-85d0-5f79dc99b712(1.2.3.4)
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Exit code: 1
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Stdout: ''
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server Stderr: 'volume set: failed: Another transaction is in progress for manila-glusterfs-vol. Please try again after some time.\n'.
Sep 15 21:56:59.982718 ubuntu-focal-rax-dfw-0026515424 manila-share[78397]: ERROR oslo_messaging.rpc.server
Additional comments at https:/