[RBD] Volume can't be removed if other volumes were created from a snapshot of this volume and were not removed

[Sofia Enriquez]

Description of problem:
It is not possible to remove a volume e.g. test-pvc if that volume was used to
* create a snapshot e.g. test-pvc-snap (that was removed later)
* create a volume from that snapshot e.g. test-pvc-from-snap

Until you remove the volume from that snapshot (test-pvc-from-snap).
openstack client returns 0 but cinder-volume logs returns:

2021-08-03 13:09:28.616 135 DEBUG cinder.volume.drivers.rbd [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] volume has no backup snaps _delete_backup_snaps /usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py:1038
2021-08-03 13:09:28.621 135 ERROR cinder.volume.manager [req-d3dabd8a-4dff-4b0f-8145-10fa303e7980 ebcbf8b5fdde4763ba119de20b84d4f7 d8efce877a93437d8e15d11ec19e3d0e - default default] Unable to delete busy volume.: cinder.exception.VolumeIsBusy: deleting volume volume-22a31673-e685-46ef-9656-382c8dca8357 that has snapshot

Cinder Version-Release: current version and
* ceph osd set-require-min-compat-client mimic

Steps to Reproduce:

Create a volume:
openstack volume create --size 1 test-pvc

Create a snapshot:
openstack volume snapshot create --volume test-pvc test-snapshot-pvc

Create a volume from the snapshot:
openstack volume create --snapshot test-snapshot-pvc test-pvc-clone

Try deleting the initial volume, it fails as expected:
openstack volume delete test-pvc

Delete the snapshot:
openstack volume snapshot delete test-snapshot-pvc

Now, retry to delete the initial volume: it doesn't report any error:
openstack volume delete test-pvc

But the volume is still here and available:
openstack volume list

Actual results:
openstackclient returns 0 but the volume isn't removed, an ERROR is seen in the logs.

Expected results:
The volume should be removed.

[Sofia Enriquez (lsofia-enriquez)]

I guess we are going to need https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/806264 if we'd like to add CI coverage for this.

[Sofia Enriquez (lsofia-enriquez)]

Fix proposed upstream https://review.opendev.org/c/openstack/cinder/+/805887

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/cinder/+/810402

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/805887
Committed: https://opendev.org/openstack/cinder/commit/ac2f8a4d895b41ef107f88b7d22e954ed156e093
Submitter: "Zuul (22348)"
Branch: master

commit ac2f8a4d895b41ef107f88b7d22e954ed156e093
Author: Sofia Enriquez <email address hidden>
Date: Tue Aug 24 21:02:31 2021 +0000

    RBD: Call trash operation when plain deletion fails

    Currently RBD doesn't allow deleting volumes with snapshots or volume
    dependencies. This causes Cinder API errors on delete calls that should
    succeed.

    When using the RBD v2 clone api, deleting a volume that has a snapshot
    in the trash space raises a busy exception.

    In order to solve this, this patch removes the proactive VolumeIsBusy
    exception raise and calls the trash operation which should succeed when
    the volume has dependencies.

    In addition to this code it's important to enable the Ceph Trash auto
    purge. Otherwise Ceph may end up with a couple of images in trash
    namespace for a while. However, this approach is the lesser of 2 evils
    because the user will be able to delete volumes with dependencies
    while the operator could check the trash namespace and manually purge
    the images. It is definitely better to potentially trouble 1 person
    (operator) that didn't read the release notes once than troubling
    every single user.

    Closes-Bug: #1941815
    Co-Author: Eric Harney <email address hidden>
    Change-Id: I5dbbcca780017b358600016afca8a9424aa137fd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/810402
Committed: https://opendev.org/openstack/cinder/commit/5226a13d149ffd991744f37618a06267eadb5f49
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 5226a13d149ffd991744f37618a06267eadb5f49
Author: Sofia Enriquez <email address hidden>
Date: Tue Aug 24 21:02:31 2021 +0000

    RBD: Call trash operation when plain deletion fails

    Currently RBD doesn't allow deleting volumes with snapshots or volume
    dependencies. This causes Cinder API errors on delete calls that should
    succeed.

    When using the RBD v2 clone api, deleting a volume that has a snapshot
    in the trash space raises a busy exception.

    In order to solve this, this patch removes the proactive VolumeIsBusy
    exception raise and calls the trash operation which should succeed when
    the volume has dependencies.

    In addition to this code it's important to enable the Ceph Trash auto
    purge. Otherwise Ceph may end up with a couple of images in trash
    namespace for a while. However, this approach is the lesser of 2 evils
    because the user will be able to delete volumes with dependencies
    while the operator could check the trash namespace and manually purge
    the images. It is definitely better to potentially trouble 1 person
    (operator) that didn't read the release notes once than troubling
    every single user.

    Closes-Bug: #1941815
    Co-Author: Eric Harney <email address hidden>
    Change-Id: I5dbbcca780017b358600016afca8a9424aa137fd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/811151

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 19.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/811151
Committed: https://opendev.org/openstack/cinder/commit/50f4dd044508f3ff9701e1e17ccd65a89db6a7aa
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 50f4dd044508f3ff9701e1e17ccd65a89db6a7aa
Author: Sofia Enriquez <email address hidden>
Date: Tue Aug 24 21:02:31 2021 +0000

    RBD: Call trash operation when plain deletion fails

    Currently RBD doesn't allow deleting volumes with snapshots or volume
    dependencies. This causes Cinder API errors on delete calls that should
    succeed.

    When using the RBD v2 clone api, deleting a volume that has a snapshot
    in the trash space raises a busy exception.

    In order to solve this, this patch removes the proactive VolumeIsBusy
    exception raise and calls the trash operation which should succeed when
    the volume has dependencies.

    In addition to this code it's important to enable the Ceph Trash auto
    purge. Otherwise Ceph may end up with a couple of images in trash
    namespace for a while. However, this approach is the lesser of 2 evils
    because the user will be able to delete volumes with dependencies
    while the operator could check the trash namespace and manually purge
    the images. It is definitely better to potentially trouble 1 person
    (operator) that didn't read the release notes once than troubling
    every single user.

    Closes-Bug: #1941815
    Co-Author: Eric Harney <email address hidden>
    Change-Id: I5dbbcca780017b358600016afca8a9424aa137fd
    (cherry picked from commit 5226a13d149ffd991744f37618a06267eadb5f49)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/822939

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Eric Harney <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/822939
Reason: Will revisit this after fixing up RBD deletion issues in master.