migrate in-use volume results in residual src volume in storage and dest volume with "attaching" status

[Trump.Zhang]

The Nova hypervisor driver is libvirt/qemu.

Set up multi-backend (LVM and NFS) Cinder-volume with the same backend_name, then migrate in-use volume from one backend to the other, the results as follows:

1. Log of cinder-volume will contains error info:

Failed to delete migration source vol 04012d12-bf16-4159-a247-2193a72cd201: Volume 04012d12-bf16-4159-a247-2193a72cd201 is still attached, detach volume first.

And it will cause the src volume be residual after migration.

2. After migration, the volume's status will keep "attaching". Indeed, checking the instance configuration with "virsh dumpxml" comamnd, the src volume has already swapped by dest volume.

I have some analysis about it, the information as follows:

1. When migrating in-use volume, cinder will first create a new volume as the dest volume, and then call Nova API (update_server_volume) to make the hypervisior do the migrate work

2. After Nova has done the migrate work, it will call Cinder API (migrate_volume_completion). Then Cinder will try to delete the src volume from backend storage, but will failed because the volume'status is "in-use". Then Cinder will just log the error info and leave the src volume be residual:

::
#cinder.volume.manager.VolumeManager:migrate_volume_completion
       # Delete the source volume (if it fails, don't fail the migration)
        try:
            self.delete_volume(ctxt, volume_id)
        except Exception as ex:
            msg = _("Failed to delete migration source vol %(vol)s: %(err)s")
            LOG.error(msg % {'vol': volume_id, 'err': ex})
::

3. Then Cinder will copy the info of dest volume to src volume in database, and then destroy the dest volume record in database:

::
#cinder.volume.manager.VolumeManager:migrate_volume_completion
       self.db.finish_volume_migration(ctxt, volume_id, new_volume_id)
        self.db.volume_destroy(ctxt, new_volume_id)
::

4. After migrate_volume_completion returns, Nova will call Cinder API (attach_volume) to update the dest volume status with the "new_volume_id" as the request param (nova.compute.manager.ComputeManager:_swap_volume). However, the "new_volume" has already been destroyed in step 3, so Cinder will return 404, and Nova will log the exception, which results in the volume's status keep "attaching".

[Avishay Traeger (avishay-il)]

I think the solution is to allow delete_volume to succeed in this case, by checking volume_ref['migration_status'].

Could you please try the following fix:
In delete_volume, change:
if volume_ref['attach_status'] == "attached":
To:
if volume_ref['attach_status'] == "attached" and not volume_ref['migration_status']:

[haruka tanizawa (h-tanizawa)]

Hi, Trump.Zhang.
If you don't mind, can I fix this with the way of Avishay's suggestion ?

[haruka tanizawa (h-tanizawa)]

> Set up multi-backend (LVM and NFS) Cinder-volume with the same backend_name, then migrate in-use volume from one
> backend to the other, the results as follows:
Is it OK that you say about 'cinder migrate' not 'nova (live-)migrate' ? (just making sure... )

[Trump.Zhang (zhangleiqiang)]

@ h-tanizawa

Thanks for your attention and it's OK you can fix it.

P.S. Nova may also be involved if fixing this bug.

 > Is it OK that you say about 'cinder migrate' not 'nova (live-)migrate' ? (just making sure... )

Yes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100195

[haruka tanizawa (h-tanizawa)]

Can I set this as duplicate of https://bugs.launchpad.net/cinder/+bug/1316079 ?
It looks better patch than my patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by haruka tanizawa (<email address hidden>) on branch: master
Review: https://review.openstack.org/100195
Reason: https://review.openstack.org/#/c/101932/

is more appropriate.