HNAS driver fails on manage-snapshot with concurrency

[Rodrigo Barbieri]

In some CI runs, it is possible to see the error below when attempting to manage snapshots:

2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server [req-ce590a8d-e6f9-4e68-a766-d639d2bd83f2 e3808aa39190435e8ca352c4e9e05006 aa9d65b6b0e141689099519e08d17d4f - - -] Exception during message handling
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 167, in wrapped
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 519, in wrapper
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/manager.py", line 2251, in manage_snapshot
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server driver_options)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/driver.py", line 1172, in manage_existing_snapshot
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server if not self.hnas.check_snapshot(snapshot['provider_location']):
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 316, in check_snapshot
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server self._execute(command)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/utils.py", line 459, in _wrapper
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server return r.call(f, *args, **kwargs)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 206, in call
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server return attempt.get(self._wrap_exception)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server six.reraise(self.value[0], self.value[1], self.value[2])
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/share/drivers/hitachi/hnas/ssh.py", line 525, in _execute
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server check_exit_code=True)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 524, in ssh_execute
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd)
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server Command: ssc 127.0.0.1 console-context --evs 15 path-to-object-number -f FS-ManilaCI2 /snapshots/ed92ebe4-733b-4903-8e76-213e643ab445/1c59780a-1323-483e-a51f-2614c5b055d8
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server Exit code: 1
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server Stdout: u'Unable to run path-to-object-number as path-to-object-number is currently running on volume 39\n'
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server Stderr: u''
2017-01-26 07:51:07.997 7045 ERROR oslo_messaging.rpc.server

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/426806

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/426806
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=229515178494688f70f57d5ec29629ee3377373d
Submitter: Jenkins
Branch: master

commit 229515178494688f70f57d5ec29629ee3377373d
Author: Alyson Rosa <email address hidden>
Date: Mon Jan 30 10:27:53 2017 -0200

    HNAS: Fix concurrency error when managing snapshots

    During manage snapshot operation, if HNAS is already running a command
    'path-to-object-number' it may cause concurrency issues, since HNAS
    can run only one 'path-to-object-number' command at a time.

    Fixing it by adding a retry when this message is returned.

    Change-Id: If0f0b2d6f7e4ba3203d10c549181a3d31113624e
    Closes-bug: #1660288

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.0.0rc1 release candidate.