In the zfs job migration tests are frequently failing and the manila share log shows corresponding zfs command failures where the backend resource cannot be found. For example in https://review.openstack.org/#/c/636819/ manila_tempest_tests.tests.api.admin.test_migration.MigrationWithShareShrinkingNFSTest, test_shrink_on_migrated_share_2_False[backend,positive], and MigrationWithShareShrinkingNFSTest all failed, and manila-share log contains these errors:
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-45cb3661-0aef-44c9-a699-f933db65cfab None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs set sharenfs=off alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot unshare 'alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653': not found: unshare(1M) failed\n"
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2986, in delete_share_instance
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server exception=excep)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2962, in delete_share_instance
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 283, in update_access_rules
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 322, in _update_access_rules
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 390, in _update_rules_through_share_driver
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 138, in wrap
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 687, in update_access
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server executor=executor)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 48, in wrapped_func
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 46, in source_func
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 301, in update_access
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.zfs("set", rules_str, dataset_name)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 147, in zfs
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self.execute('sudo', 'zfs', *cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs set sharenfs=off alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot unshare 'alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653': not found: unshare(1M) failed\n"
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-2311f17d-5155-4c6d-a90c-977723c67265 None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs set sharenfs=off alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot unshare 'alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39': not found: unshare(1M) failed\n"
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2986, in delete_share_instance
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server exception=excep)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2962, in delete_share_instance
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 283, in update_access_rules
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 322, in _update_access_rules
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 390, in _update_rules_through_share_driver
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 138, in wrap
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 687, in update_access
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server executor=executor)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 48, in wrapped_func
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 46, in source_func
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 301, in update_access
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.zfs("set", rules_str, dataset_name)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 147, in zfs
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self.execute('sudo', 'zfs', *cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs set sharenfs=off alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot unshare 'alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39': not found: unshare(1M) failed\n"
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-5508b42e-7297-47ec-990b-b3bf21b0986e None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs get used betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot open 'betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6': dataset does not exist\n"
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2400, in manage_share
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_instance, driver_options)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 715, in manage_existing
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server consumed_space = self.get_zfs_option(old_dataset_name, "used")
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 143, in get_zfs_option
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._get_option(dataset_name, option_name, False, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 115, in _get_option
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server 'sudo', app, 'get', option_name, resource_name, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs get used betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot open 'betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6': dataset does not exist\n"
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server
Setting this to INVALID since we haven't been able to reproduce it in ten runs with current code via https:/ /review. opendev. org/#/c/ 670339. We can re-open if this starts happening again.