Can create orphan snapshots via rapid, concurrent create and delete snapshot operations

[Patrick Crews]

NOTE: Observed testing via devstack and high concurrency stress testing.

I have observed many instances where cinder snapshots are created, but end up in an orphan / zombie state.
They continue to be listed as part of snapshot-list and count against user quota, but they cannot be deleted, either by the user or the admin.

This generally happens when one user creates the snapshot and a second user issues a delete_snapshot command within a reasonably quick timeframe (max elapsed time observed is 16 seconds between create and delete calls).

Have also noted such snapshots are referenced in the mysql slow log.
Any guidance / requests for additional debugging, logs, etc are welcome.

pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-list
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| 459dc788-9ade-4211-a559-4fe0bcec8ad2 | e7d48279-c499-4efc-900b-e6867b10e6ff | creating | snapshot-0x7f4709178ad0-73 | 1 |
| 717c38bf-d05d-42fa-8a70-ce46270f9bd1 | b64918c8-e75b-4fd0-ac54-9b2b2236f418 | creating | snapshot-0x7f4709178ad0-27 | 99 |
| 890ad93b-9f44-437b-ac50-f10b2df0b9cb | f00415bb-aa6d-4f4a-a4eb-8ce43d234d00 | creating | snapshot-0x7f470890bc50-2 | 1 |
| 8a722581-d5b1-47b2-afad-4360cca516a0 | cca63364-cad7-4025-97e0-0b2b388a472e | creating | snapshot-0x7f470890bc50-70 | 1 |
| b286c7ae-be84-441b-ad7c-1a1e68ba14ce | d1a5f0cf-b9e7-4458-9103-9be3e0feba14 | creating | snapshot-0x7f470890bc50-59 | 1 |
| d378d2ff-f997-4b21-8a24-30ee8b613688 | b0cfd1ca-9d49-4401-b3f2-73aed760033e | creating | snapshot-0x7f4709178e90-9 | 2 |
| f14d89cd-733a-4dfe-8e81-f0e55be38cb7 | a6a683a7-b83c-4705-8ee8-0207669b27bf | creating | snapshot-0x7f470915c410-78 | 1 |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-delete 459dc788-9ade-4211-a559-4fe0bcec8ad2
Delete for snapshot 459dc788-9ade-4211-a559-4fe0bcec8ad2 failed: Bad Request (HTTP 400) (Request-ID: req-da0daed4-49ff-49da-922d-db8c81810ea6)
ERROR: Unable to delete any of the specified snapshots.

[Patrick Crews (patrick-crews)]

API calls with timing data for several of the snapshots referenced in the initial report
Each 0x7f4709895b50 identifier - references a unique user

41915:[2014-11-20 13:45:34,025] erlking-dev/INFO/stdout: 0x7f4709895b50: create volume: volume-0x7f4709895b50-23
42893:[2014-11-20 13:45:55,553] erlking-dev/INFO/stdout: 0x7f4709178ad0: create snapshot: snapshot-0x7f4709178ad0-73
43217:[2014-11-20 13:46:06,492] erlking-dev/INFO/stdout: 0x7f470915c410: delete snapshot: 459dc788-9ade-4211-a559-4fe0bcec8ad2

2433:[2014-11-20 13:25:07,516] erlking-dev/INFO/stdout: 0x7f470890bc50: create volume: volume-0x7f470890bc50-3
3679:[2014-11-20 13:25:40,910] erlking-dev/INFO/stdout: 0x7f470890bc50: create snapshot: snapshot-0x7f470890bc50-2
3949:[2014-11-20 13:25:56,585] erlking-dev/INFO/stdout: 0x7f4709178ad0: delete snapshot: 890ad93b-9f44-437b-ac50-f10b2df0b9cb

4029:[2014-11-20 13:25:58,303] erlking-dev/INFO/stdout: 0x7f4709178e90: create volume: volume-0x7f4709178e90-3
6185:[2014-11-20 13:27:07,529] erlking-dev/INFO/stdout: 0x7f4709178e90: create snapshot: snapshot-0x7f4709178e90-9
6311:[2014-11-20 13:27:10,649] erlking-dev/INFO/stdout: 0x7f470890bc50: delete snapshot: d378d2ff-f997-4b21-8a24-30ee8b613688

[Jay Bryant (jsbryant)]

Have you tried using 'cinder reset-state' to get the snapshots to a state where where you can try to delete them?

[Patrick Crews (patrick-crews)]

reset-state fails for both project owner and admin accounts:
pcrews@erlking-dev:~/git/rannsaka$ cinder --os-username=demo --os-password=OVERNUMEROUSNESS --os-tenant-name=demo --os-auth-url=http://192.168.0.5:5000 snapshot-list
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| 1e278f43-8796-4619-850b-edde6329ff98 | be3802e7-92da-482b-99c4-f5eaf11e1edf | creating | snapshot-0x7f3f5e4785d0-12 | 1 |
| 51827ebc-5562-479a-bf66-038dec6a9d41 | ccbc6fbb-92b8-4fc1-b69d-0bccaba5db20 | creating | snapshot-0x7f3f5ecbf850-39 | 1 |
| 52734450-0b1d-4552-9fb7-5adc378d8201 | be3802e7-92da-482b-99c4-f5eaf11e1edf | creating | snapshot-0x7f3f5ef21f10-11 | 1 |
| 7fdb72a0-76fd-4ae2-bd92-1f92b44fc0a6 | 80755bb2-2de8-475f-9705-0f7688cd979a | creating | snapshot-0x7f3f5ef21f10-44 | 99 |
| 96da8c45-0c4e-406b-9da7-4c85f67105dd | ce33c298-35c8-467e-84f6-643b11217235 | creating | snapshot-0x7f3f5e4785d0-85 | 99 |
| d015f3f8-1c15-409b-b302-0ddf6de5aded | 02b6b1dc-a37e-4422-a1d6-2048568cdeb5 | creating | snapshot-0x7f3f5e4785d0-25 | 1 |
| e3df7bbc-a122-4a20-8cc3-aef63fbeb146 | e65ab1e7-5968-4d60-b929-24bee143c75b | deleting | snapshot-0x7f3f5e464690-11 | 1 |
| e5a1f054-44f0-4e4d-8024-00127bd669cb | df8c6bf5-ba7d-488f-a15d-5feac62cd65f | creating | snapshot-0x7f3f5e4785d0-21 | 99 |
| f18e595a-ed29-4555-8617-0b08fafc4d4d | 4cf4a1cc-a2dd-40e9-ac76-8a5525649fb3 | creating | snapshot-0x7f3f5e464690-28 | 1 |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+

pcrews@erlking-dev:~/git/rannsaka$ cinder --os-username=demo --os-password=OVERNUMEROUSNESS --os-tenant-name=demo --os-auth-url=http://192.168.0.5:5000 snapshot-delete 1e278f43-8796-4619-850b-edde6329ff98
Delete for snapshot 1e278f43-8796-4619-850b-edde6329ff98 failed: Bad Request (HTTP 400) (Request-ID: req-9a0b246f-5275-4419-af08-b09773c0d1b3)
ERROR: Unable to delete any of the specified snapshots.

pcrews@erlking-dev:~/git/rannsaka$ cinder --os-username=demo --os-password=OVERNUMEROUSNESS --os-tenant-name=demo --os-auth-url=http://192.168.0.5:5000 reset-state 1e278f43-8796-4619-850b-edde6329ff98
Reset state for volume 1e278f43-8796-4619-850b-edde6329ff98 failed: Not Found (HTTP 404) (Request-ID: req-f66c2681-40a1-4481-a235-5315c9241033)
ERROR: Unable to reset the state for the specified volume(s).

pcrews@erlking-dev:~/git/rannsaka$ cinder --os-username=admin --os-password=OVERNUMEROUSNESS --os-tenant-name=admin --os-auth-url=http://192.168.0.5:5000 reset-state 1e278f43-8796-4619-850b-edde6329ff98
Reset state for volume 1e278f43-8796-4619-850b-edde6329ff98 failed: Not Found (HTTP 404) (Request-ID: req-f62f869e-6235-4cf5-80af-91c4950a052f)
ERROR: Unable to reset the state for the specified volume(s).

[Xin， Xiaohui (xiaohui-xin)]

@pcrews
Does this issue still exist? I'm interested in this. Do you have detailed scenario to help me reproducing it? I have tried to create/delete snapshots with two users(demo/admin), but just found admin can delete snapshots created by user demo, and demo user cannot delete the snapshot created by admin. Thanks!