rbd snapshot delete fails if backend is missing file

[Dirk Mueller]

When snapshots are stored in rbd, but the rados backend does not actually server the snapshot anymore (because it was for example deleted behind's cinder back), its not possible to remove the snapshot information from cinder anymore. because it does not handle the rbd.ImageNotFound exception raised by the rbd bindings. every attempt will end up in "error_deleting".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/151257

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151257
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c776607c4305304f56e779403e1d23ad51e8dcb5
Submitter: Jenkins
Branch: master

commit c776607c4305304f56e779403e1d23ad51e8dcb5
Author: Dirk Mueller <email address hidden>
Date: Thu Jan 29 14:11:35 2015 +0100

    RBD: Make snapshot_delete more robust

    Since there is no equivalent of a force-snapshot-delete,
    the normal snapshot-delete should be more robust to
    also handle backend errors. In case the backend does not have
    the image snapshot anymore, log an info message and succeed the
    operation.

    Change-Id: I7fe0878dbc07053ac78272b6998513fafa1c36e8
    Closes-Bug: #1415905
    Related-Bug: #1361926

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/cinder 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299989

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/300433

[Saverio Proto (zioproto)]

Hello,
this bug is still open for me.
The proposed patch fixes the action of unprotecting the snapshot, but when you actually go for delete the snapshot you get this stacktrace

2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 179, in lso_inner1
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher return lso_inner2(inst, context, snapshot, **kwargs)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 178, in lso_inner2
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher return f(*_args, **_kwargs)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 725, in delete_snapshot
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher snapshot.save()
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 715, in delete_snapshot
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher self.driver.delete_snapshot(snapshot)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-04-01 11:38:35.551 17573 TRACE oslo_messaging.rpc....

[Guangyu Suo (yugsuo)]

Yes, this error still exists, for the ImageNotFound situation, it only logs the info, and then to remove the snapshot from backend, this will raise the exception:

2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 771, in delete_snapshot
2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher volume.remove_snap(snap_name)
2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/rbd.py", line 552, in remove_snap
2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher raise make_ex(ret, 'error removing snapshot %s from %s' % (name, self.name))
2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error removing snapshot snapshot-5773f327-2ba0-4399-b1db-8da2eb8471ea from volume-6daf5bf5-d1bb-458d-a5d6-93534e802985
2016-04-08 17:06:04.746 18588 ERROR oslo_messaging.rpc.dispatcher

and result the snapshot in error_deleting status. So it is better to return if snapshot does not exist in backend.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Saverio Proto (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/299989
Reason: I will try this again in Mitaka when it is merged in master

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300433
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=677eb1c4160c08cfce2900495741f0ea15f566fa
Submitter: Jenkins
Branch: master

commit 677eb1c4160c08cfce2900495741f0ea15f566fa
Author: Saverio Proto <email address hidden>
Date: Fri Apr 1 13:50:16 2016 +0200

    RBD: delete snapshots if missing in the backend

    snapshot delete should be more robust to also handle backend errors.
    In case the backend does not have the image snapshot anymore,
    log an info message and succeed the operation.
    A first attempt to fix this problem was found on
    https://review.openstack.org/#/c/151257/
    This patch is already merged but it does not fix the problem
    completely.

    Change-Id: I1e80c2f71b79a5d40a0b7a42f03eb5bf911c24f0
    Closes-Bug: #1415905

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/306604

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/306610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/306604
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4c26a6dd2657715389dcc5bb3ebbfb5e35e249d8
Submitter: Jenkins
Branch: stable/mitaka

commit 4c26a6dd2657715389dcc5bb3ebbfb5e35e249d8
Author: Saverio Proto <email address hidden>
Date: Fri Apr 1 13:50:16 2016 +0200

    RBD: delete snapshots if missing in the backend

    snapshot delete should be more robust to also handle backend errors.
    In case the backend does not have the image snapshot anymore,
    log an info message and succeed the operation.
    A first attempt to fix this problem was found on
    https://review.openstack.org/#/c/151257/
    This patch is already merged but it does not fix the problem
    completely.

    Change-Id: I1e80c2f71b79a5d40a0b7a42f03eb5bf911c24f0
    Closes-Bug: #1415905
    (cherry picked from commit 677eb1c4160c08cfce2900495741f0ea15f566fa)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 9.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/cinder 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/306610
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

[James Page (james-page)]

Hello Dirk, or anyone else affected,

Accepted cinder into liberty-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:liberty-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-liberty-needed to verification-liberty-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-liberty-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package cinder - 2:7.0.2-0ubuntu1~cloud1
---------------

 cinder (2:7.0.2-0ubuntu1~cloud1) trusty-liberty; urgency=medium
 .
   * RBD: Delete snapshots if missing in the Ceph backend (LP: #1415905):
     - d/p/cinder-306610.patch: Apply patch from review 306610.