Temporary volume accepts deletion while it is used

Bug #1970768 reported by Hironori Shiina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Low
Unassigned

Bug Description

A temporary volume is created when an attached volume is backed up. This temporary volume can be deleted by DELETE API because its status is 'available'.

Deleting a LUN while it is connected causes unexpected errors in a volume backend. Depending on a volume backend, this can cause a serious problem which may affect other users.

This problem actually happened once in our production system by misoperation. As a result, some processes handling iSCSI devices such as a scsi command[1] got stuck. In cinder-volume, a thread which called the process also got stuck with holding a lock[2]. This blocked any operations requiring this lock.

[1] https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L81
[2] https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L865

Steps to reproduce:
1. Create a volume:
   $ openstack volume create --size=20 myvolume
2. Attach the volume to a server:
   $ openstack server add volume myvolume myserver --device /dev/vdb
3. Backup the volume:
   $ openstack volume backup create myvolume --force
4. List volumes during backing up and find a temporary volume:
   $ openstack volume list
   +--------------------------------------+-------------------------------------------------+------------+------+---------------------------------------------------------------+
   | ID | Name | Status | Size | Attached to |
   +--------------------------------------+-------------------------------------------------+------------+------+---------------------------------------------------------------+
   | 287b0cee-4ca2-41d4-8e98-d6d705751851 | backup-vol-c146c1fc-864d-4269-b369-dc5bfffca42f | available | 20 | |
   | c146c1fc-864d-4269-b369-dc5bfffca42f | myvolume | backing-up | 20 | Attached to 78ce3358-364d-4500-b122-4bbef72b99cd on /dev/vdb |
   +--------------------------------------+-------------------------------------------------+------------+------+---------------------------------------------------------------+
5. Delete the temporary volume:
   $ openstack volume delete backup-vol-c146c1fc-864d-4269-b369-dc5bfffca42f

Changed in cinder:
status: New → In Progress
Changed in cinder:
importance: Undecided → Low
tags: added: backup-service temporary-volume
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/826949
Committed: https://opendev.org/openstack/cinder/commit/53c13891b313d58923b223f6fbb1882e991f2785
Submitter: "Zuul (22348)"
Branch: master

commit 53c13891b313d58923b223f6fbb1882e991f2785
Author: Hironori Shiina <email address hidden>
Date: Fri Jan 28 21:38:03 2022 -0500

    Prevent temporary volume from being deleted accidentally

    A temporary volume can be deleted while it is in use by DELETE API
    accidentally because its status is `available`. To avoid this deletion,
    this fix sets a value which doesn't accept deletion to volume status of
    a temporary volume. When a temporary volume is used for backing up,
    `backing-up` is set. When a temporary volume is used for reverting a
    snapshot, `in-use` is set because the volume is attached by a host.

    Closes-Bug: #1970768
    Change-Id: Ib6a2e4d68e532b91161df5245c17ce815f12f935

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/cinder/+/840730

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/840730
Committed: https://opendev.org/openstack/cinder/commit/d773fae8256f76ee78c81a5c07096a45227c6569
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit d773fae8256f76ee78c81a5c07096a45227c6569
Author: Hironori Shiina <email address hidden>
Date: Fri Jan 28 21:38:03 2022 -0500

    Prevent temporary volume from being deleted accidentally

    A temporary volume can be deleted while it is in use by DELETE API
    accidentally because its status is `available`. To avoid this deletion,
    this fix sets a value which doesn't accept deletion to volume status of
    a temporary volume. When a temporary volume is used for backing up,
    `backing-up` is set. When a temporary volume is used for reverting a
    snapshot, `in-use` is set because the volume is attached by a host.

    Closes-Bug: #1970768
    Change-Id: Ib6a2e4d68e532b91161df5245c17ce815f12f935
    (cherry picked from commit 53c13891b313d58923b223f6fbb1882e991f2785)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/831186
Committed: https://opendev.org/openstack/cinder/commit/cf2f3d07ae4eccb4371d031fef992d107e9c07c4
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cf2f3d07ae4eccb4371d031fef992d107e9c07c4
Author: Hironori Shiina <email address hidden>
Date: Fri Jan 28 21:38:03 2022 -0500

    Prevent temporary volume from being deleted accidentally

    A temporary volume can be deleted while it is in use by DELETE API
    accidentally because its status is `available`. To avoid this deletion,
    this fix sets a value which doesn't accept deletion to volume status of
    a temporary volume. When a temporary volume is used for backing up,
    `backing-up` is set. When a temporary volume is used for reverting a
    snapshot, `in-use` is set because the volume is attached by a host.

    Closes-Bug: #1970768
    Change-Id: Ib6a2e4d68e532b91161df5245c17ce815f12f935
    (cherry picked from commit 53c13891b313d58923b223f6fbb1882e991f2785)
    (cherry picked from commit d773fae8256f76ee78c81a5c07096a45227c6569)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/831187
Committed: https://opendev.org/openstack/cinder/commit/553264535dc34f676ff4db3c77e1869c1fb3c6b7
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 553264535dc34f676ff4db3c77e1869c1fb3c6b7
Author: Hironori Shiina <email address hidden>
Date: Fri Jan 28 21:38:03 2022 -0500

    Prevent temporary volume from being deleted accidentally

    A temporary volume can be deleted while it is in use by DELETE API
    accidentally because its status is `available`. To avoid this deletion,
    this fix sets a value which doesn't accept deletion to volume status of
    a temporary volume. When a temporary volume is used for backing up,
    `backing-up` is set. When a temporary volume is used for reverting a
    snapshot, `in-use` is set because the volume is attached by a host.

    Closes-Bug: #1970768
    Change-Id: Ib6a2e4d68e532b91161df5245c17ce815f12f935
    (cherry picked from commit 53c13891b313d58923b223f6fbb1882e991f2785)
    (cherry picked from commit d773fae8256f76ee78c81a5c07096a45227c6569)
    (cherry picked from commit cf2f3d07ae4eccb4371d031fef992d107e9c07c4)
    Conflicts:
            cinder/volume/manager.py

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 20.0.1

This issue was fixed in the openstack/cinder 20.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 19.1.1

This issue was fixed in the openstack/cinder 19.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 21.0.0.0rc1

This issue was fixed in the openstack/cinder 21.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 18.2.1

This issue was fixed in the openstack/cinder 18.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/victoria)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/831188
Reason: stable/victoria branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/victoria if you want to further work on this patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.