Manila does not delete snapshots that are reported to be busy

Bug #1564623 reported by Goutham Pacha Ravi
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
NidhiMittalHada

Bug Description

As of commit: https://github.com/openstack/manila/commit/dfabad3 if the snapshot-delete API is used while using snapshots of ZFSonLinux or NetApp backends, if the driver reports that the snapshot is busy, manila seems to be resetting the status to `available`.

tags: added: netapp
tags: added: zfsonlinux
Changed in manila:
assignee: nobody → Goutham Pacha Ravi (gouthamr)
assignee: Goutham Pacha Ravi (gouthamr) → nobody
tags: added: low-hanging-fruit
Changed in manila:
assignee: nobody → NidhiMittalHada (nidhimittal19)
Changed in manila:
status: New → In Progress
Revision history for this message
NidhiMittalHada (nidhimittal19) wrote :
Download full text (3.9 KiB)

Steps to reproduce:-
-----------------------------------
1) manila snapshot-create --name snapshot1 --description "Snapshot1" 726245d0-72c3-400d-b3ee-b3bc40e1207e

2)stack@controller:~/devstack$ manila snapshot-list
+--------------------------------------+--------------------------------------+-----------+-----------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+-----------+-----------+------------+
| 3d8406c2-d6ed-4559-a9b8-199bd9933444 | 726245d0-72c3-400d-b3ee-b3bc40e1207e | available | snapshot1 | 1 |
+--------------------------------------+--------------------------------------+-----------+-----------+------------+

3)stack@controller:~/devstack$ manila --debug snapshot-delete 3d8406c2-d6ed-4559-a9b8-199bd9933444

exception raised!!!!

ShareSnapshotIsBusy: DemoVol1/share_d061ea16_fb95_40ee_9d23_524ebdefcdcd@share_snapshot_849700f8_4f6b_4a78_a4e2_67863a84c3b9

4)stack@controller:~/devstack$ manila snapshot-list
+--------------------------------------+--------------------------------------+-----------+-----------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+-----------+-----------+------------+
| 3d8406c2-d6ed-4559-a9b8-199bd9933444 | 726245d0-72c3-400d-b3ee-b3bc40e1207e | available | snapshot1 | 1 |
+--------------------------------------+--------------------------------------+-----------+-----------+------------+
stack@controller:~/devstack$

snapshot has become "available".

but if we do
stack@controller:~/devstack$ manila --debug snapshot-force-delete 3d8406c2-d6ed-4559-a9b8-199bd9933444

msg comes as
The driver reported that the snapshot 3d8406c2-d6ed-4559-a9b8-199bd9933444 was busy on the backend. Since this operation was forced, the snapshot will be deleted from Manila's database. A cleanup on the backend may be necessary.

and.....
stack@controller:~/devstack$ manila snapshot-list
+----+----------+--------+------+------------+
| ID | Share ID | Status | Name | Share Size |
+----+----------+--------+------+------------+
+----+----------+--------+------+------------+

snapshot is deleted from manila db.
=======================================================================================================

As per code :-
------------------
       try:
            self.driver.delete_snapshot(context, snapshot_instance,
                                        share_server=share_server)
        except exception.ShareSnapshotIsBusy:
            with excutils.save_and_reraise_exception() as exc:
                if force:
                    msg = _("The driver reported that the snapshot %s "
                            "was busy on the backend. Since this "
                            "operation was forced, the snapshot will "
                            "be deleted from Manila's database. A "
                            "cleanup on the backend may be necessary.")
                    LOG.ex...

Read more...

Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

The behavior should be that, no matter what the exception is that is raised by the driver in the delete path, set the status field to 'error' (not 'available' as was being done in this case)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/311996

Revision history for this message
NidhiMittalHada (nidhimittal19) wrote :

https://review.openstack.org/#/c/311996/1/manila/share/manager.py

As per review comments here, there is "NOT" one opinion on what status of snapshot should we keep,
when snapshot is returned to be busy by backend.

There are two arguments :-

status should be "ERROR_DELETING"
1)no matter what the exception is that is raised by the driver in the delete path, set the status field to 'error' (not 'available' as was being done in the other exception cases)

status should be "AVAILABLE"
2)If the driver export "ShareSnapshotIsBusy", it means snapshot are being used. not the snapshot itself is a problem.
If the snapshot state is set to available, The snapshot can continue to be used. User can continue to operate this snapshot.
If the snapshot state is set to error_deleting, only the administrator can operate this snapshot.
so, Is it the available state more suitable than error_deleting state?

Need to decide on this ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/311996
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=69eaebc774c4f2132f6dc123ec296df7d3bd0cb0
Submitter: Jenkins
Branch: master

commit 69eaebc774c4f2132f6dc123ec296df7d3bd0cb0
Author: nidhimittalhada <email address hidden>
Date: Tue May 3 14:24:01 2016 +0530

    Delete Snapshot: status wrongly set when busy

    This fix sets the snapshot status correctly to 'error_deleting',
    when snapshot reported to be busy by backend driver.

    Change-Id: Iecfd32b8265ce9f7cdfcba6d89397f269024b75d
    Closes-Bug: #1564623

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/manila 3.0.0.0b1

This issue was fixed in the openstack/manila 3.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/354812

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/mitaka)

Reviewed: https://review.openstack.org/354812
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=34773c372c73a8ce8edd97a59e2969cca93dc1da
Submitter: Jenkins
Branch: stable/mitaka

commit 34773c372c73a8ce8edd97a59e2969cca93dc1da
Author: nidhimittalhada <email address hidden>
Date: Tue May 3 14:24:01 2016 +0530

    Delete Snapshot: status wrongly set when busy

    This fix sets the snapshot status correctly to 'error_deleting',
    when snapshot reported to be busy by backend driver.

    Change-Id: Iecfd32b8265ce9f7cdfcba6d89397f269024b75d
    Closes-Bug: #1564623
    (cherry picked from commit 69eaebc774c4f2132f6dc123ec296df7d3bd0cb0)

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.