Garbage remains when attached volume was migrated with nfs driver

Bug #1391172 reported by Rikimaru Honjo
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Rushil Chugh
Juno
Fix Released
Undecided
Unassigned

Bug Description

[Environments]
I use juno-stable(2014.2) and nfs-driver.

[Bug]
I executed volume-migration API for attached-volume (status is "in-use").
In this case, volume was migrated from nfs-backend host to other nfs-backend host.

Result of above API execution, volume migration process was completed.
However, following error log was generated.
And, volume file remained on original host.

Error Log:
2014-10-24 16:09:37.593 15148 ERROR cinder.volume.manager [req-6f2b2936-5856-49d7-859b-1a3a5542504d 7e25f622315c443fb264f9827fae9083 d01b0b96236d41c3a0022bb951f71c14 - - -] Failed to delete migration source vol 7d0a1407-0238-45c8-93f0-49a163c65503: Volume 7d0a1407-0238-45c8-93f0-49a163c65503 is still attached, detach volume first.

[Cause of bug]
Volume migration deletes original volume at final stage of process.
But, volume migration does not detach except when retype occurs.
Therefore volume deletion is failed("in-use" volume can not be deleted).
Error log and remained file is result of fail of deletion.

Rushil Chugh (rushil)
Changed in cinder:
assignee: nobody → Rushil Chugh (rushil)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143431

Changed in cinder:
status: New → In Progress
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
Rushil Chugh (rushil) wrote :

The root cause of the bug is that status_update['status'] is only set for retype and not for migration from one NFS backend to another NFS backend. The status_update['status'] in https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1114 updates the current volume status for retyping scenarios thus updating the status to "in-use". This causes the if condition in https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1134 to be true leading to the detach of the volume. The value of status_update['status'] needs to be updated as it is getting reused in lines https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1149 and https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1156. In the retyping case https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1134, https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1149 and https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1156 hold true as the status_update['status'] gets updated thus causing the migration to succeed. Hence, this migration issue doesn't arise for NFS to iSCSI migration and iSCSI to NFS migration.
But, in the scenario of migration from a NFS backend to another NFS backend, status_update['status'] doesn't get updated and https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1135 doesn't get called as the current volume status is "none". That is the root cause of why migration fails when an attached NFS volume is being migrated to another NFS backend.
https://review.openstack.org/#/c/143431/ tries to fix it by making a status_update = {'status': self._get_original_status(volume)} call just before https://github.com/openstack/cinder/blob/master/cinder/volume/manager.py#L1134.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/143431
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c2023be0056f0a0fdc141115d5fca0cd4a164ca5
Submitter: Jenkins
Branch: master

commit c2023be0056f0a0fdc141115d5fca0cd4a164ca5
Author: Rushil Chugh <email address hidden>
Date: Mon Dec 8 17:09:36 2014 -0500

    Garbage Remains when Attached Volume is Migrated with NFS Driver

    Garbage volume remained when an attached NFS volume was migrated
    to another NFS backend. An error message was generated as the
    attached volume was not detached before being migrated. The root
    cause of the issue is that status_update['status'] is only set
    for retype and not for migration from one NFS backend to another
    NFS backend. This patch proposes to fix the aforementioned issue
    where migration fails from one NFS backend to another NFS backend.

    Change-Id: I657a46d98b1e2a52cec128308cbbf39166c3625b
    Closes-bug: 1391172

Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/147996

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/juno)

Reviewed: https://review.openstack.org/147996
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0b2f27813314bc7b6cd3ae8510a5dff55d0d3d90
Submitter: Jenkins
Branch: stable/juno

commit 0b2f27813314bc7b6cd3ae8510a5dff55d0d3d90
Author: Rushil Chugh <email address hidden>
Date: Thu Jan 15 14:12:30 2015 -0500

    Garbage Remains when Attached Volume is Migrated with NFS Driver

    Garbage volume remained when an attached NFS volume was migrated
    to another NFS backend. An error message was generated as the
    attached volume was not detached before being migrated. The root
    cause of the issue is that status_update['status'] is only set
    for retype and not for migration from one NFS backend to another
    NFS backend. This patch proposes to fix the aforementioned issue
    where migration fails from one NFS backend to another NFS backend.

    Change-Id: I657a46d98b1e2a52cec128308cbbf39166c3625b
    Closes-bug: 1391172
    (cherry picked from commit: c2023be0056f0a0fdc141115d5fca0cd4a164ca5)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in cinder:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.