Fail to delete a volume which is once attached by LVM backend and LIO

Bug #1410566 reported by Mitsuhiro Tanino
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Mitsuhiro Tanino
Juno
Fix Released
High
Matthew Booth

Bug Description

In LVM iSCSI driver with 'iscsi_helper = lioadm' (after transition to Target Objects), volume deletion which is once attached fails because the volume is in-use status.

LOG:
------
2015-01-13 15:34:52.951 INFO cinder.volume.targets.lio [req-4935b10c-6dc8-44b0-8712-bc99437ba5aa 130e1342922842dbb955643b0b447f96 4f38320ef3fb48c49cf2d0f85307207c] Skipping remove_export. No iscsi_target provisioned for volume: 4221d57d-0110-4e94-8088-5fbe0a169726
.....
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvremove --config activation { retry_deactivation = 1} devices { ignore_suspended_devices = 1} -f stack-vg2/volume-4221d57d-0110-4e94-8088-5fbe0a169726
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher Exit code: 5
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2015-01-13 15:35:03.608 TRACE oslo.messaging.rpc.dispatcher Stderr: u' Logical volume stack-vg2/volume-4221d57d-0110-4e94-8088-5fbe0a169726 in use.\n'
------

Steps to reproduce:
  (a) Boot a nova instance
  (b) Create a volume using LVM backend and LIO
  (c) Attach a volume to an instance
  (d) Detach a volume from instance
  (e) Delete the volume after attached

=> Fail to delete the volume

After transitioning to the new driver and target model, iscsi_targets is not added to the table during create_export() phase. However, remove_export() in LIO target is still reffering empty iscsi_targets table. This causes NotFound exception and remove_export() skips to do remove_iscsi_target().

As a result, iscsi target is not removed and the target continues to grab the volume(logical volume) as an in-use status.

Revision history for this message
Mitsuhiro Tanino (mitsuhiro-tanino) wrote :
Changed in cinder:
assignee: nobody → Mitsuhiro Tanino (mitsuhiro-tanino)
status: New → In Progress
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/148038

Eric Harney (eharney)
Changed in cinder:
importance: Undecided → High
milestone: none → kilo-2
Mike Perez (thingee)
Changed in cinder:
milestone: kilo-2 → kilo-3
Changed in cinder:
assignee: Mitsuhiro Tanino (mitsuhiro-tanino) → John Griffith (john-griffith)
Changed in cinder:
assignee: John Griffith (john-griffith) → Mitsuhiro Tanino (mitsuhiro-tanino)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/148038
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=fb8bfea97b54f3d0971fdc1b0473e8a60b1badb3
Submitter: Jenkins
Branch: master

commit fb8bfea97b54f3d0971fdc1b0473e8a60b1badb3
Author: Mitsuhiro Tanino <email address hidden>
Date: Fri Jan 16 22:04:17 2015 -0500

    Refactoring for export functions in Target object

    Currently, export functions such as create_export() are implemented
    in individual Target code, but most of them are same and these are
    common features in each target.
    This patch moves these methods to parent ISCSITarget class to
    commonalize and each Target simply inherit these methods from parent
    class. As a result of this change, LioAdm can inherit ISCSITarget
    class directly instead of inheriting TgtAdm class.
    This simplifies dependency of targets and improves maintainability.

    By refactoring these methods, this patch also fixes following issues.

    (a) Fix bug #1410566
    After transitioning to the new driver and target model, iscsi_targets
    is not added to the table during create_export() phase.
    However, remove_export() in LIO Target is still reffering empty
    iscsi_targets table. This causes NotFound exception and remove_export()
    skips to do remove_iscsi_target().
    As a result, iscsi target is not removed and the target continues to
    grab the volume(logical volume) as an in-use status.
    This patch fix the problem.

    (b) Re-export a volume with CHAP
    Current Tgt Target recreate iscsi target without CHAP during
    ensure_export() even if the volume is exported with CHAP previously.
    This patch changes this bahaviour to recreate iscsi target using
    previous CHAP which is stored in volume file on state_path dir.

    Closes-Bug: 1410566
    Change-Id: Iea3d94e35a4ced4dafc1b61e2df6b075cf200577

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/175349

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/juno)

Reviewed: https://review.openstack.org/175349
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=89a6082d089facb79a7c74d6a0d8c3853d9046f9
Submitter: Jenkins
Branch: stable/juno

commit 89a6082d089facb79a7c74d6a0d8c3853d9046f9
Author: Matthew Booth <email address hidden>
Date: Wed Apr 15 14:50:08 2015 +0100

    Fix re-export of iscsi volume when using lioadm

    This change ensures we check for existing chap credentials in the db
    when re-exporting a volume. Without this change we overwrite the
    existing credentials, resulting in an inaccessible volume.

    This is a manual cut and paste of portions of change
    Iea3d94e35a4ced4dafc1b61e2df6b075cf200577. Specifically, we add the
    overridden _get_target_chap_auth for LioAdm, and add a context
    argument to the function throughout.

    It is not possible or desirable to backport the whole change for several
    reasons:

    * It is based on a refactor which is not on this branch
    * It includes a refactor which is not relevant to this branch
    * It includes more than 1 fix

    Closes-bug: #1410566
    Change-Id: I9859306b8beb0e1d8e3b3f6464c3f42e672e3949

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.