[zfsonlinux] job migration tests are failing

Bug #1822415 reported by Tom Barron
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Invalid
Undecided
Unassigned

Bug Description

In the zfs job migration tests are frequently failing and the manila share log shows corresponding zfs command failures where the backend resource cannot be found. For example in https://review.openstack.org/#/c/636819/ manila_tempest_tests.tests.api.admin.test_migration.MigrationWithShareShrinkingNFSTest, test_shrink_on_migrated_share_2_False[backend,positive], and MigrationWithShareShrinkingNFSTest all failed, and manila-share log contains these errors:

Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-45cb3661-0aef-44c9-a699-f933db65cfab None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs set sharenfs=off alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot unshare 'alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653': not found: unshare(1M) failed\n"
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2986, in delete_share_instance
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server exception=excep)
Mar 28 09:42:21.266288 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2962, in delete_share_instance
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 283, in update_access_rules
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 322, in _update_access_rules
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 390, in _update_rules_through_share_driver
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 138, in wrap
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 687, in update_access
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server executor=executor)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 48, in wrapped_func
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Mar 28 09:42:21.268382 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 46, in source_func
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 301, in update_access
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.zfs("set", rules_str, dataset_name)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 147, in zfs
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self.execute('sudo', 'zfs', *cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs set sharenfs=off alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot unshare 'alpha/manila_share_177342ae_7ec0_404a_9ad9_49c559b65653': not found: unshare(1M) failed\n"
Mar 28 09:42:21.270333 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-2311f17d-5155-4c6d-a90c-977723c67265 None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs set sharenfs=off alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot unshare 'alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39': not found: unshare(1M) failed\n"
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2986, in delete_share_instance
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server exception=excep)
Mar 28 09:42:21.273106 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2962, in delete_share_instance
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 283, in update_access_rules
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 322, in _update_access_rules
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/access.py", line 390, in _update_rules_through_share_driver
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_server=share_server
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 138, in wrap
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 687, in update_access
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server executor=executor)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 48, in wrapped_func
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return source_func(self, *args, **kwargs)
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Mar 28 09:42:21.275257 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 46, in source_func
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 301, in update_access
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server self.zfs("set", rules_str, dataset_name)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 147, in zfs
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self.execute('sudo', 'zfs', *cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs set sharenfs=off alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot unshare 'alpha/manila_share_b7cb6f0f_53ef_4a70_8c19_eff5f0bd2e39': not found: unshare(1M) failed\n"
Mar 28 09:42:21.277057 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server [None req-5508b42e-7297-47ec-990b-b3bf21b0986e None None] Exception during message handling: ProcessExecutionError: Unexpected error while running command.
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Command: sudo zfs get used betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Exit code: 1
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stdout: u''
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: Stderr: u"cannot open 'betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6': dataset does not exist\n"
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 186, in wrapped
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 568, in wrapper
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2400, in manage_share
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server share_instance, driver_options)
Mar 28 09:45:41.865711 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/driver.py", line 715, in manage_existing
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server consumed_space = self.get_zfs_option(old_dataset_name, "used")
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 143, in get_zfs_option
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return self._get_option(dataset_name, option_name, False, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 115, in _get_option
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server 'sudo', app, 'get', option_name, resource_name, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/zfsonlinux/utils.py", line 97, in execute
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server return executor(*cmd, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/ganesha/utils.py", line 79, in __call__
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ret = processutils.ssh_execute(ssh, cmd, **kwargs)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 570, in ssh_execute
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Command: sudo zfs get used betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Exit code: 1
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stdout: u''
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server Stderr: u"cannot open 'betta/subdir/manila_share_9786d938_3d8d_4b04_a439_c431998ed7f6': dataset does not exist\n"
Mar 28 09:45:41.867529 ubuntu-bionic-rax-dfw-0004453309 manila-share[12297]: ERROR oslo_messaging.rpc.server

Tom Barron (tpb)
tags: added: driver zfsonlinux
summary: - [zfs] job migration tests are failing
+ [zfsonlinux] job migration tests are failing
Revision history for this message
Tom Barron (tpb) wrote :

Setting this to INVALID since we haven't been able to reproduce it in ten runs with current code via https://review.opendev.org/#/c/670339. We can re-open if this starts happening again.

Changed in manila:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.