| 2014-11-21 01:12:54 |
Patrick Crews |
description |
NOTE: Observed testing via devstack and high concurrency stress testing.
I have observed many instances where cinder snapshots are created, but end up in an orphan / zombie state.
They continue to be listed as part of snapshot-list and count against user quota, but they cannot be deleted, either by the user or the admin.
This generally happens when one user creates the snapshot and a second user issues a delete_snapshot command within a reasonably quick timeframe (max elapsed time observed is 16 seconds between create and delete calls).
Have also noted such snapshots are referenced in the mysql slow log.
pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-list
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| 459dc788-9ade-4211-a559-4fe0bcec8ad2 | e7d48279-c499-4efc-900b-e6867b10e6ff | creating | snapshot-0x7f4709178ad0-73 | 1 |
| 717c38bf-d05d-42fa-8a70-ce46270f9bd1 | b64918c8-e75b-4fd0-ac54-9b2b2236f418 | creating | snapshot-0x7f4709178ad0-27 | 99 |
| 890ad93b-9f44-437b-ac50-f10b2df0b9cb | f00415bb-aa6d-4f4a-a4eb-8ce43d234d00 | creating | snapshot-0x7f470890bc50-2 | 1 |
| 8a722581-d5b1-47b2-afad-4360cca516a0 | cca63364-cad7-4025-97e0-0b2b388a472e | creating | snapshot-0x7f470890bc50-70 | 1 |
| b286c7ae-be84-441b-ad7c-1a1e68ba14ce | d1a5f0cf-b9e7-4458-9103-9be3e0feba14 | creating | snapshot-0x7f470890bc50-59 | 1 |
| d378d2ff-f997-4b21-8a24-30ee8b613688 | b0cfd1ca-9d49-4401-b3f2-73aed760033e | creating | snapshot-0x7f4709178e90-9 | 2 |
| f14d89cd-733a-4dfe-8e81-f0e55be38cb7 | a6a683a7-b83c-4705-8ee8-0207669b27bf | creating | snapshot-0x7f470915c410-78 | 1 |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-delete 459dc788-9ade-4211-a559-4fe0bcec8ad2
Delete for snapshot 459dc788-9ade-4211-a559-4fe0bcec8ad2 failed: Bad Request (HTTP 400) (Request-ID: req-da0daed4-49ff-49da-922d-db8c81810ea6)
ERROR: Unable to delete any of the specified snapshots. |
NOTE: Observed testing via devstack and high concurrency stress testing.
I have observed many instances where cinder snapshots are created, but end up in an orphan / zombie state.
They continue to be listed as part of snapshot-list and count against user quota, but they cannot be deleted, either by the user or the admin.
This generally happens when one user creates the snapshot and a second user issues a delete_snapshot command within a reasonably quick timeframe (max elapsed time observed is 16 seconds between create and delete calls).
Have also noted such snapshots are referenced in the mysql slow log.
Any guidance / requests for additional debugging, logs, etc are welcome.
pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-list
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
| 459dc788-9ade-4211-a559-4fe0bcec8ad2 | e7d48279-c499-4efc-900b-e6867b10e6ff | creating | snapshot-0x7f4709178ad0-73 | 1 |
| 717c38bf-d05d-42fa-8a70-ce46270f9bd1 | b64918c8-e75b-4fd0-ac54-9b2b2236f418 | creating | snapshot-0x7f4709178ad0-27 | 99 |
| 890ad93b-9f44-437b-ac50-f10b2df0b9cb | f00415bb-aa6d-4f4a-a4eb-8ce43d234d00 | creating | snapshot-0x7f470890bc50-2 | 1 |
| 8a722581-d5b1-47b2-afad-4360cca516a0 | cca63364-cad7-4025-97e0-0b2b388a472e | creating | snapshot-0x7f470890bc50-70 | 1 |
| b286c7ae-be84-441b-ad7c-1a1e68ba14ce | d1a5f0cf-b9e7-4458-9103-9be3e0feba14 | creating | snapshot-0x7f470890bc50-59 | 1 |
| d378d2ff-f997-4b21-8a24-30ee8b613688 | b0cfd1ca-9d49-4401-b3f2-73aed760033e | creating | snapshot-0x7f4709178e90-9 | 2 |
| f14d89cd-733a-4dfe-8e81-f0e55be38cb7 | a6a683a7-b83c-4705-8ee8-0207669b27bf | creating | snapshot-0x7f470915c410-78 | 1 |
+--------------------------------------+--------------------------------------+----------+----------------------------+------+
pcrews@erlking-dev:~/git/rannsaka$ cinder snapshot-delete 459dc788-9ade-4211-a559-4fe0bcec8ad2
Delete for snapshot 459dc788-9ade-4211-a559-4fe0bcec8ad2 failed: Bad Request (HTTP 400) (Request-ID: req-da0daed4-49ff-49da-922d-db8c81810ea6)
ERROR: Unable to delete any of the specified snapshots. |
|
