Can create orphan snapshots via rapid, concurrent create and delete snapshot operations
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
New
|
Undecided
|
Unassigned |
Bug Description
NOTE: Observed testing via devstack and high concurrency stress testing.
I have observed many instances where cinder snapshots are created, but end up in an orphan / zombie state.
They continue to be listed as part of snapshot-list and count against user quota, but they cannot be deleted, either by the user or the admin.
This generally happens when one user creates the snapshot and a second user issues a delete_snapshot command within a reasonably quick timeframe (max elapsed time observed is 16 seconds between create and delete calls).
Have also noted such snapshots are referenced in the mysql slow log.
Any guidance / requests for additional debugging, logs, etc are welcome.
pcrews@
+------
| ID | Volume ID | Status | Display Name | Size |
+------
| 459dc788-
| 717c38bf-
| 890ad93b-
| 8a722581-
| b286c7ae-
| d378d2ff-
| f14d89cd-
+------
pcrews@
Delete for snapshot 459dc788-
ERROR: Unable to delete any of the specified snapshots.
API calls with timing data for several of the snapshots referenced in the initial report
Each 0x7f4709895b50 identifier - references a unique user
41915:[2014-11-20 13:45:34,025] erlking- dev/INFO/ stdout: 0x7f4709895b50: create volume: volume- 0x7f4709895b50- 23 dev/INFO/ stdout: 0x7f4709178ad0: create snapshot: snapshot- 0x7f4709178ad0- 73 dev/INFO/ stdout: 0x7f470915c410: delete snapshot: 459dc788- 9ade-4211- a559-4fe0bcec8a d2
42893:[2014-11-20 13:45:55,553] erlking-
43217:[2014-11-20 13:46:06,492] erlking-
2433:[2014-11-20 13:25:07,516] erlking- dev/INFO/ stdout: 0x7f470890bc50: create volume: volume- 0x7f470890bc50- 3 dev/INFO/ stdout: 0x7f470890bc50: create snapshot: snapshot- 0x7f470890bc50- 2 dev/INFO/ stdout: 0x7f4709178ad0: delete snapshot: 890ad93b- 9f44-437b- ac50-f10b2df0b9 cb
3679:[2014-11-20 13:25:40,910] erlking-
3949:[2014-11-20 13:25:56,585] erlking-
4029:[2014-11-20 13:25:58,303] erlking- dev/INFO/ stdout: 0x7f4709178e90: create volume: volume- 0x7f4709178e90- 3 dev/INFO/ stdout: 0x7f4709178e90: create snapshot: snapshot- 0x7f4709178e90- 9 dev/INFO/ stdout: 0x7f470890bc50: delete snapshot: d378d2ff- f997-4b21- 8a24-30ee8b6136 88
6185:[2014-11-20 13:27:07,529] erlking-
6311:[2014-11-20 13:27:10,649] erlking-