Huawei FC driver fails to create a volume from a snapshot with Dorado V6 - Bad or unexpected response from the storage volume backend API: Create luncopy error

[Nobuto Murata]

More specifically with the following Tempest test:
tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot

- OpenStack Ussuri
- Huawei Dorado 6000 V6

We are testing Cinder Huawei FC driver with Dorado 6000 V6 storage array, and found the following problem. At this point, we are using Ussuri, but we're pretty sure the issue can be reproduced with the latest version of Cinder too since there is no significant update around the code path.

How to reproduce:
1. deploy OpenStack with Cinder
2. integrate Huawei Dorado by writing the config

[[cinder.conf]]
enabled_backends = cinder-huawei
[cinder-huawei]
volume_driver = cinder.volume.drivers.huawei.huawei_driver.HuaweiFCDriver
cinder_huawei_conf_file = /etc/cinder/cinder_huawei_conf.xml
volume_backend_name = oceanstor
use_multipath_for_image_xfer = True
enforce_multipath_for_image_xfer = True

[[cinder_huawei_conf.xml]]
<config>
    <Storage>
        <Product>Dorado</Product>
        <Protocol>FC</Protocol>
        <UserName>HIDDEN</UserName>
        <UserPassword>HIDDEN</UserPassword>
        <RestURL>https://FQDN:8088/deviceManager/rest/</RestURL>
    </Storage>
    <LUN>
        <LUNType>Thin</LUNType>
        <StoragePool>StoragePool001</StoragePool>
        <LUNCopySpeed>3</LUNCopySpeed>
        <LUNCloneMode>fastclone</LUNCloneMode>
        <HyperSyncSpeed>3</HyperSyncSpeed>
    </LUN>
    <FC>
        <Initiator Name="None" ALUA="1" FAILOVERMODE="1" PATHTYPE="0" />
    </FC>
</config>

3. create a volume

$ openstack volume create --size 10 test-volume
-> confirm status gets "available"

4. create a snapshot

$ openstack volume snapshot create --volume test-volume test-volume-snapshot
-> confirm status gets "available"

5. create a new volume from the snapshot

$ openstack volume create --snapshot test-volume-snapshot volume-from-snapshot

Expected:
step 5 just works.

Actual:
step 5 fails with the following error as the bottom line:

/var/log/cinder/cinder-volume.log:
re-scheduling SchedulerAPI.create_volume attempt 3 due to Bad or unexpected response from the storage volume backend API: Create luncopy error.

and the following API call:

INFO cinder.volume.drivers.huawei.rest_client [...]

Request URL: https://FQDN:8088/deviceManager/rest/ARRAY_ID/luncopy

Call Method: POST

Request Data: {'TYPE': 219, 'NAME': '17398032-188edd03d5a80a50163572', 'DESCRIPTION': '17398032-188edd03d5a80a50163572', 'COPYSPEED': '2', 'LUNCOPYTYPE':
'1', 'SOURCELUN': 'INVALID;284;INVALID;INVALID;INVALID', 'TARGETLUN': 'INVALID;293;INVALID;INVALID;INVALID'}

Response Data:{'data': {}, 'error': {'code': 50331651, 'description': 'The entered parameter is incorrect.', 'suggestion': 'Enter a correct parameter.'}}

[Nobuto Murata (nobuto)]

I initially thought 'INVALID;284;INVALID;INVALID;INVALID' is wrong, but it looks like a red herring since those are actually from the expected function def create_luncopy.
https://github.com/openstack/cinder/blob/086f619cee654690d66314ac792a12c2799022a1/cinder/volume/drivers/huawei/rest_client.py#L361-L386

[Nobuto Murata (nobuto)]

I've got some input from a different channel that the luncopy method shouldn't be used with Dorado V6, and V6 is not supported by the driver in the OpenStack Cinder project. But Dorado V6 support for both iSCSI and FC are called out explicitly just for the record.

https://docs.openstack.org/releasenotes/cinder/ussuri.html#relnotes-16-0-0-stable-ussuri-new-features
> New Features
>
> Huawei Cinder Driver Support Dorado V6 Storage.(iSCSI, FC)

[huangbingyan (bingyanh)]

Hello, if the huawei cinder driver needs to fix this problem,
Is it necessary to merge into the branches? Ussuri or what

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/818778

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/819790

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/819790
Committed: https://opendev.org/openstack/cinder/commit/77da18785af7bede26d3478934864e293c6f4ef6
Submitter: "Zuul (22348)"
Branch: master

commit 77da18785af7bede26d3478934864e293c6f4ef6
Author: bingyan <email address hidden>
Date: Tue Nov 30 18:01:08 2021 +0800

    fix 'huawei dorado v6' create new volume from snapshot error

    change the luncopy mode,in dorado v6 we use clone_pair
    modified some interface to call dorado v6

    Closes-Bug: #1942342
    Change-Id: If26ebd6a94512b1d8ecc1b786f518f667fc19ffe

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/830050

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinder/+/834961

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/837051

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/837052

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/830050
Reason: Ussuri is going EOL [0] and the cinder project team has decided to abandon all open patches on stable/ussuri [1].
[0] https://<email address hidden>/thread/UJ7Y76LRWTZQSSDAGMPWCCVVGGIX3TOG/
[1] https://meetings.opendev.org/meetings/cinder/2023/cinder.2023-12-13-14.00.log.html#l-66

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/cinder/+/818778
Reason: Ussuri is going EOL [0] and the cinder project team has decided to abandon all open patches on stable/ussuri [1].
[0] https://<email address hidden>/thread/UJ7Y76LRWTZQSSDAGMPWCCVVGGIX3TOG/
[1] https://meetings.opendev.org/meetings/cinder/2023/cinder.2023-12-13-14.00.log.html#l-66

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodrigo Barbieri <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinder/+/834961

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/837052
Reason: stable/victoria branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/837051
Reason: stable/wallaby branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/wallaby if you want to further work on this patch.