Volume creation from snapshot is failing using VNX driver

[Peter Wang]

Reviewing both volume creation failure event logs, it seems the failure is reported by the driver due to migration failure of volumes internal to EMC storage.

Volume c66dabbd-3efe-4687-9baf-b58c8930ce58:
-------------------------------------------------------------------------------------------------------------------------------
Oct 25 11:02:31 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:02:31.475 671024 INFO cinder.volume.flows.manager.create_volume [req-197e3ca1-b5f5-43eb-8853-8bf3e00d132e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] Volume c66dabbd-3efe-4687-9baf-b58c8930ce58: being created as snap with specification: {'status': u'creating', 'volume_size': 20, 'volume_name': 'volume-c66dabbd-3efe-4687-9baf-b58c8930ce58', 'snapshot_id': '855c1f6c-422b-4c5d-96e3-db7ef6d82a18'}

Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.601 671024 ERROR cinder.volume.drivers.emc.vnx.taskflows [req-197e3ca1-b5f5-43eb-8853-8bf3e00d132e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] Failed to migrate volume between source vol 63590 and dest vol 350.<<<<<<<<<<<<<
Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.606 671024 WARNING cinder.volume.drivers.emc.vnx.taskflows [req-197e3ca1-b5f5-43eb-8853-8bf3e00d132e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] MigrateLunTask.revert: cleanup migration session: 63590 -> 350.
Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.607 671024 INFO storops.vnx.navi_command [req-197e3ca1-b5f5-43eb-8853-8bf3e00d132e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] call command: /opt/Navisphere/bin/naviseccli -h 10.149.24.138 -user admin -password *** -scope global migrate -list -source 63590

-------------------------------------------------------------------------------------------------------------------------------
Volume f158b482-6079-4de1-80b6-352a605f8a5f:
-------------------------------------------------------------------------------------------------------------------------------
Oct 25 11:02:35 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:02:35.480 671024 INFO cinder.volume.flows.manager.create_volume [req-0df66af5-6803-4e19-8abb-fbec74f51e4e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] Volume f158b482-6079-4de1-80b6-352a605f8a5f: being created as snap with specification: {'status': u'creating', 'volume_size': 1, 'volume_name': 'volume-f158b482-6079-4de1-80b6-352a605f8a5f', 'snapshot_id': '0877de43-494a-447d-9f4e-2b6bfb3da425'}

Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.663 671024 ERROR cinder.volume.drivers.emc.vnx.taskflows [req-0df66af5-6803-4e19-8abb-fbec74f51e4e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] Failed to migrate volume between source vol 63592 and dest vol 353.<<<<<<<<<<<<<<<<
Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.668 671024 WARNING cinder.volume.drivers.emc.vnx.taskflows [req-0df66af5-6803-4e19-8abb-fbec74f51e4e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] MigrateLunTask.revert: cleanup migration session: 63592 -> 353.
Oct 25 11:06:03 overcloud-controller-prod6b-1 cinder-volume: 2017-10-25 15:06:03.669 671024 INFO storops.vnx.navi_command [req-0df66af5-6803-4e19-8abb-fbec74f51e4e 208df083edfd446d888879ae60391d1a 12990b3d3d4f41eeb3a52e7ce4a85dce - default default] call command: /opt/Navisphere/bin/naviseccli -h 10.149.24.138 -user admin -password *** -scope global migrate -list -source 63592
-------------------------------------------------------------------------------------------------------------------------------

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/520006
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=552d6fd33a4a2534eea6ca71765ab9936f5920b7
Submitter: Zuul
Branch: master

commit 552d6fd33a4a2534eea6ca71765ab9936f5920b7
Author: Peter Wang <email address hidden>
Date: Wed Nov 15 04:13:28 2017 -0500

    VNX: Fix cloning failure when system is busy

    In newton, the 30s wait was removed due to refactor, in this case,
    If the session hasn't started in the VNX, the driver would perform
    unexpected rollback for the migration.

    Adding a wait will make sure the session is started on the VNX thus
    cloning can succeed.

    Change-Id: I3419e628d42eddfe19c55cd72e7b9469e9fccc94
    Closes-bug: #1732462

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/531532

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/531532
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=41c565debb918c07f18fbe864002abcc241fe50e
Submitter: Zuul
Branch: stable/pike

commit 41c565debb918c07f18fbe864002abcc241fe50e
Author: Peter Wang <email address hidden>
Date: Wed Nov 15 04:13:28 2017 -0500

    VNX: Fix cloning failure when system is busy

    In newton, the 30s wait was removed due to refactor, in this case,
    If the session hasn't started in the VNX, the driver would perform
    unexpected rollback for the migration.

    Adding a wait will make sure the session is started on the VNX thus
    cloning can succeed.

    Change-Id: I3419e628d42eddfe19c55cd72e7b9469e9fccc94
    Closes-bug: #1732462
    (cherry picked from commit 552d6fd33a4a2534eea6ca71765ab9936f5920b7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.1.0 release.