HDS_HNAS: Cannot create shares from snapshots created from managed shares

[Rodrigo Barbieri]

The following error is displayed when attempting to create a share from a snapshot created from a managed share. Inspecting the code it seems the managed share ID is not retrieved from private storage for this operation, or not used properly when creating the snapshot.

2016-05-20 15:40:02.172 DEBUG manila.share.drivers.hitachi.ssh [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939a
a6] Command ssc 127.0.0.1 console-context --evs 3 tree-clone-job-submit -e -f FS-ManilaDev1 /snapshots/fdb14194-3551-4882-bc39-81cac09a28b7/dce21940-fd4b-433b-9dee-3f08
313c5265 /shares/cdb887be-08e2-4c21-a325-62b0b58883ee result: out = - err = Source path: Cannot access "/snapshots/fdb14194-3551-4882-bc39-81cac09a28b7".
tree-clone-job-submit: Request failed: Source path not found
 - exit = 1. from (pid=11082) _execute /opt/stack/manila/manila/share/drivers/hitachi/ssh.py:375
2016-05-20 15:40:02.173 ERROR manila.share.drivers.hitachi.ssh [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939a
a6] Error running SSH command.
2016-05-20 15:40:02.176 ERROR manila.share.drivers.hitachi.ssh [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939a
a6] Unexpected error while running command.
Command: ssc 127.0.0.1 console-context --evs 3 tree-clone-job-submit -e -f FS-ManilaDev1 /snapshots/fdb14194-3551-4882-bc39-81cac09a28b7/dce21940-fd4b-433b-9dee-3f08313
c5265 /shares/cdb887be-08e2-4c21-a325-62b0b58883ee
Exit code: 1
Stdout: u''
Stderr: u'Source path: Cannot access "/snapshots/fdb14194-3551-4882-bc39-81cac09a28b7".\ntree-clone-job-submit: Request failed: Source path not found\n'
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh Traceback (most recent call last):
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/opt/stack/manila/manila/share/drivers/hitachi/ssh.py", line 110, in tree_clone
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh output, err = self._execute(command)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/opt/stack/manila/manila/utils.py", line 602, in _wrapper
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh return r.call(f, *args, **kwargs)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 206, in call
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh return attempt.get(self._wrap_exception)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 247, in get
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh six.reraise(self.value[0], self.value[1], self.value[2])
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/usr/local/lib/python2.7/dist-packages/retrying.py", line 200, in call
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/opt/stack/manila/manila/share/drivers/hitachi/ssh.py", line 359, in _execute
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh check_exit_code=True)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 512, in ssh_execute
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh cmd=sanitized_cmd)
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh ProcessExecutionError: Unexpected error while running command.
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh Command: ssc 127.0.0.1 console-context --evs 3 tree-clone-job-submit -e -f FS-ManilaDev1 /snapshots/fdb14194-3551-4882-bc39-81cac09a28b7/dce21940-fd4b-433b-9dee-3f08313c5265 /shares/cdb887be-08e2-4c21-a325-62b0b58883ee
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh Exit code: 1
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh Stdout: u''
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh Stderr: u'Source path: Cannot access "/snapshots/fdb14194-3551-4882-bc39-81cac09a28b7".\ntree-clone-job-submit: Request failed: Source path not found\n'
2016-05-20 15:40:02.176 TRACE manila.share.drivers.hitachi.ssh
2016-05-20 15:40:02.179 ERROR manila.share.manager [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939aa6] Share instance cdb887be-08e2-4c21-a325-62b0b58883ee failed on creation.
2016-05-20 15:40:02.180 WARNING manila.share.manager [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939aa6] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.
2016-05-20 15:40:02.235 ERROR oslo_messaging.rpc.server [req-4df14b50-a38c-4956-a8f4-30789ec3d522 ca71c4cc3fed4a1cbbfd97f968f7d536 234b54ea84124dbb8fb897aa83939aa6] Exception during handling message
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 153, in dispatch
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 146, in wrapped
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server return f(self, *args, **kwargs)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/utils.py", line 616, in wrapper
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server return func(self, *args, **kwargs)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 1035, in create_share_instance
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server {'status': constants.STATUS_ERROR}
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server self.force_reraise()
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/manager.py", line 1003, in create_share_instance
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server share_server=share_server)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/hitachi/hds_hnas.py", line 274, in create_share_from_snapshot
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server path = self._create_share_from_snapshot(share, snapshot)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/hitachi/hds_hnas.py", line 645, in _create_share_from_snapshot
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server self.hnas.tree_clone(src_path, dest_path)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server File "/opt/stack/manila/manila/share/drivers/hitachi/ssh.py", line 119, in tree_clone
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server raise exception.HNASBackendException(msg=msg)
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server HNASBackendException: HNAS Backend Exception: Unexpected error while running command.
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server Command: ssc 127.0.0.1 console-context --evs 3 tree-clone-job-submit -e -f FS-ManilaDev1 /snapshots/fdb14194-3551-4882-bc39-81cac09a28b7/dce21940-fd4b-433b-9dee-3f08313c5265 /shares/cdb887be-08e2-4c21-a325-62b0b58883ee
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server Exit code: 1
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server Stdout: u''
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server Stderr: u'Source path: Cannot access "/snapshots/fdb14194-3551-4882-bc39-81cac09a28b7".\ntree-clone-job-submit: Request failed: Source path not found\n'
2016-05-20 15:40:02.235 TRACE oslo_messaging.rpc.server

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316113
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=081fc4860b7ba3a87020d2ddae006f667616fed0
Submitter: Jenkins
Branch: master

commit 081fc4860b7ba3a87020d2ddae006f667616fed0
Author: Rodrigo Barbieri <email address hidden>
Date: Fri May 13 10:23:46 2016 -0300

    Fix HDS HNAS errors caused by incorrect IDs

    When attempting to allow_access on a managed share, it fails
    because the proper share ID is not retrieved from private storage
    prior to attempting to validate that the share exists in the
    backend. The same happens when trying to create a share from a
    snapshot created from a managed share, the proper share ID is
    not retrieved from private storage. While we are dealing with
    two possible different IDs it is important to properly display
    the API share ID in log messages so it can be matched to the
    share instances ID, and not all log messages are accurately doing
    so.

    This change addresses this by retrieving the ID from
    private storage first for update_access and
    create_share_from_snapshot operations. The proper unit test changes
    included in this patch required a refactor of several IDs to ensure
    this problem is addressed in unit tests, thus it made sense to
    address several bugs caused by the same problem, having the same fix
    and requiring modifications to the same lines of code.

    Closes-bug: #1581541
    Closes-bug: #1584179
    Closes-bug: #1583785
    Change-Id: I8cdb1a8a72a4ac7e710f57e3c288d48cd2adf0dd

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/manila 3.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/325874

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/325874
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=5884b3ca3f3bc4ace14e6991bca739bec199dd9f
Submitter: Jenkins
Branch: stable/mitaka

commit 5884b3ca3f3bc4ace14e6991bca739bec199dd9f
Author: Rodrigo Barbieri <email address hidden>
Date: Mon Jun 6 09:41:55 2016 -0300

    Fix HDS HNAS errors caused by incorrect IDs

    When attempting to allow_access on a managed share, it fails
    because the proper share ID is not retrieved from private storage
    prior to attempting to validate that the share exists in the
    backend. The same happens when trying to create a share from a
    snapshot created from a managed share, the proper share ID is
    not retrieved from private storage. While we are dealing with
    two possible different IDs it is important to properly display
    the API share ID in log messages so it can be matched to the
    share instances ID, and not all log messages are accurately doing
    so.

    This change addresses this by retrieving the ID from
    private storage first for update_access and
    create_share_from_snapshot operations. The proper unit test changes
    included in this patch required a refactor of several IDs to ensure
    this problem is addressed in unit tests, thus it made sense to
    address several bugs caused by the same problem, having the same fix
    and requiring modifications to the same lines of code.

    (cherry-picked from commit 081fc4860b7ba3a87020d2ddae006f667616fed0)
    Closes-bug: #1581541
    Closes-bug: #1584179
    Closes-bug: #1583785
    Change-Id: I8cdb1a8a72a4ac7e710f57e3c288d48cd2adf0dd