HNAS concurrency issues when creating and deleting snapshots

[Rodrigo Barbieri]

As can be seen in CI, there are concurrency issues when creating snapshots. Looking at the code, the same can be said about when deleting snapshots.

2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh [req-6b295d80-0948-4117-9e24-9f6ad2f1b72b - - - - -] Failed to create directory /snapshots/0eae0b04-0ebb-41bf-9072-3bf40f9d22fd/bc0fc7f0-294b-47e0-af87-b1d6bf1fe1f5.
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh Traceback (most recent call last):
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 654, in _locked_selectfs
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh self._execute(command)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/opt/stack/new/manila/manila/utils.py", line 459, in _wrapper
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh return r.call(f, *args, **kwargs)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 206, in call
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh return attempt.get(self._wrap_exception)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh six.reraise(self.value[0], self.value[1], self.value[2])
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 623, in _execute
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh check_exit_code=True)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 524, in ssh_execute
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh cmd=sanitized_cmd)
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh ProcessExecutionError: Unexpected error while running command.
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh Command: ssc 127.0.0.1 console-context --evs 15 selectfs FS-ManilaCI2
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh ssc 127.0.0.1 console-context --evs 15 mkdir -p /snapshots/0eae0b04-0ebb-41bf-9072-3bf40f9d22fd/bc0fc7f0-294b-47e0-af87-b1d6bf1fe1f5
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh Exit code: 1
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh Stdout: u'Current selected file system: FS-ManilaCI2, number(39)\n'
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh Stderr: u'Current file system invalid: VolumeNotFound\n'
2017-02-07 17:17:49.519 20176 ERROR manila.share.drivers.hitachi.hnas.ssh
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server [req-6b295d80-0948-4117-9e24-9f6ad2f1b72b - - - - -] Exception during message handling
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 167, in wrapped
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 519, in wrapper
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2644, in create_snapshot
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server {'status': constants.STATUS_ERROR})
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2637, in create_snapshot
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server context, snapshot_instance, share_server=share_server) or {}
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/driver.py", line 394, in create_snapshot
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server export_locations = self._create_snapshot(hnas_share_id, snapshot)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/driver.py", line 993, in _create_snapshot
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server self.hnas.create_directory(dest_path)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 357, in create_directory
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server self._locked_selectfs('create', dest_path)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 658, in _locked_selectfs
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server raise exception.HNASBackendException(msg=msg)
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server HNASBackendException: HNAS Backend Exception: Failed to create directory /snapshots/0eae0b04-0ebb-41bf-9072-3bf40f9d22fd/bc0fc7f0-294b-47e0-af87-b1d6bf1fe1f5.
2017-02-07 17:17:50.208 20176 ERROR oslo_messaging.rpc.server

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/431100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/431100
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4daea8d7cceedcd10108ac70fb09b1d9fcad28f4
Submitter: Jenkins
Branch: master

commit 4daea8d7cceedcd10108ac70fb09b1d9fcad28f4
Author: Rodrigo Barbieri <email address hidden>
Date: Wed Feb 8 14:30:21 2017 -0200

    HNAS: Fix concurrency creating/deleting snapshots

    When attempting to create or delete empty folders in HNAS,
    respective to snapshot creation and deletion, there can
    be context changes due to concurrent requests received by
    HNAS.

    The result of the failed command is an error that the
    operation cannot be completed in the current filesystem if
    the context change switched to another EVS. In case the
    context switch changed only the filesystem within the same
    EVS, the operation may be run in the wrong filesystem,
    although, those operations are harmless.

    Closes-bug: #1662615
    Change-Id: I3439f8ab2dd967a438eac575172e4b96e897b46c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 5.0.0.0b1 development milestone.