EMC volume create_cloned_volume failure leaves target volume

[Samuel Matzek]

A create_cloned_volume call occurred when licenses missing, probably the license for cloning.

This call failed, but as part of the failure it created a volume in the SAN that was not cleaned up.

Stack and error:
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/emc/emc_vmax_fc.py", line 69, in create_cloned_volume
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher volpath = self.common.create_cloned_volume(volume, src_vref)
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/emc/emc_vmax_common.py", line 227, in create_cloned_volume
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher return self._create_cloned_volume(cloneVolume, sourceVolume)
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/emc/emc_vmax_common.py", line 1920, in _create_cloned_volume
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher sourceInstance)
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/emc/emc_vmax_provision.py", line 577, in create_element_replica
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher data=exceptionMessage)
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Error Create Cloned Volume: Volume: volume-boot-8233E8B_106013P-roll-the-dice-35-83c9ec2b-5c72 Source Volume:volume-MyVolume_image-eefb3aaf-e71d. Return code: 1. Error: Create Replica at step Issuing CreateReplica:StorRepGroupCloneEstablish failed: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1000010 L:2 C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1000010 The feature being requested is not currently licensed : "Establish failed" : 2 : 214 : "The feature being requested is not currently licensed" @
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher [1] com.emc.cmp.osls.se.osl.Replica.StorRepGroupCloneEstablish():1380
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher [0] com.emc.cmp.osls.se.array.job.JOB_RepCreate.run():261
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher
2014-10-22 00:49:00.974 14376 TRACE oslo.messaging.rpc.dispatcher
2014-10-22 00:49:01.127 14376 ERROR oslo.messaging._drivers.common [req-142b5917-92e3-4949-a4e5-39a69573d1cd 0 b6bbb50c2f32408b99dbe23a52faf485 - - -] Returning exception Bad or unexpected response from the storage volume backend API: Error Create Cloned Volume: Volume: volume-boot-8233E8B_106013P-roll-the-dice-35-83c9ec2b-5c72 Source Volume:volume-MyVolume_image-eefb3aaf-e71d. Return code: 1. Error: Create Replica at step Issuing CreateReplica:StorRepGroupCloneEstablish failed: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1000010 L:2 C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1000010 The feature being requested is not currently licensed : "Establish failed" : 2 : 214 : "The feature being requested is not currently licensed" @
  [1] com.emc.cmp.osls.se.osl.Replica.StorRepGroupCloneEstablish():1380
  [0] com.emc.cmp.osls.se.array.job.JOB_RepCreate.run():261
 to caller

[Carl Pecinovsky (csky)]

Xing,
Can you set a milestone for this fix? Even with the new code added for _examine_source_and_create_clone(), I don't see rollback processing in place to remove the volume if an exception occurs after it is created. Also, there is still the path in place where create_element_replica() creates the clone as part of the data copying operation. Unless you separate the calls to do that as well, I don't think the problem this defect was originally opened for can be addressed since you would not get a reference back to the orphaned volume. Not sure. Thanks.

[Xing Yang (xing-yang)]

Carl,

We'll check the license ahead of time to avoid this situation. Yes, we won't get a reference if the volume can't be created and therefore we can't delete it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/159553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159553
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=49bda16e5a57497cb6dfe0f85748a7e8dde3be9c
Submitter: Jenkins
Branch: master

commit 49bda16e5a57497cb6dfe0f85748a7e8dde3be9c
Author: Xing Yang <email address hidden>
Date: Thu Feb 26 12:27:16 2015 -0500

    Check license before clone in VMAX driver

    This patch checks if a license for clone is available before
    doing the clone operation.

    Change-Id: I69d009b2cac775c301e9a5254079238b2b2a3b10
    Closes-Bug: #1385450