tempest tests causing cinder to leave around volumes

[Sean Dague]

Running tempest full tests on a devstack environment, and eventually tests started failing. It turns out there was no longer any space in the stack-volumes vg

sudo vgs
  VG #PV #LV #SN Attr VSize VFree
  stack-volumes 1 5 0 wz--n- 5.01g 8.00m

however cinder didn't display any active volumes:

OS_USERNAME=admin cinder list
<nothing>

There were still 5 volumes in the vg:

sudo lvs
  LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
  volume-07dccc79-4d6f-42d5-9bd7-f10c7b16bc71 stack-volumes -wi-a--- 1.00g
  volume-6cccd750-420c-4a99-b919-ef15d5db1734 stack-volumes -wi-a--- 1.00g
  volume-80150244-1fed-40e4-841a-e3fa534adfc7 stack-volumes -wi-a--- 1.00g
  volume-9e755589-5a59-4df0-83d1-f9a2ec2b777b stack-volumes -wi-a--- 1.00g
  volume-bbc28867-5e23-4b15-8bc7-e768f84d9b8d stack-volumes -wi-a--- 1.00g

and the database looked as follows (csv export):

"2012-12-20 20:34:48","2012-12-20 20:34:52","2012-12-20 20:34:54","1","0789700d-766c-44e7-bc5c-38fd4b100c5e",NULL,"211f3787c603432487fe88091892463b","46dc9266a5ea4c6ca164497915122e12","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:34:49",NULL,"2012-12-20 20:34:52","volume-964202",NULL,NULL,NULL,NULL,"794eedbd-4114-45b5-b6e4-dcd959850142"
"2012-12-20 20:36:09","2012-12-20 20:36:09",NULL,"0","07dccc79-4d6f-42d5-9bd7-f10c7b16bc71",NULL,"38ed30afde30425698481960d9d33895","697dbf6004a342518ad66c0d58caa17a","blake","1","nova",NULL,NULL,NULL,"error","detached","2012-12-20 20:36:09",NULL,NULL,"volume732707",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:10:31","2012-12-20 20:17:14","2012-12-20 20:17:17","1","0f5f046e-3e54-4994-9e99-7032256278cb",NULL,"010785a6a3114b6492d54f3f3cfa39f0","d000979d31154b159966ba4eef7674b7","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:10:31",NULL,"2012-12-20 20:17:14",NULL,NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:46:07","2012-12-20 20:46:14","2012-12-20 20:46:19","1","17d08d2c-5939-494d-b682-98c15736d684",NULL,"9e6e01af8ea5450e8cc7ae0b863d4510","492eef17ebff4be79c16aa3537e2e6ae","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:46:08",NULL,"2012-12-20 20:46:14","volume-256542",NULL,NULL,NULL,NULL,"1fda8bc2-dc47-4119-8e00-5f28ecfe7f71"
"2012-12-20 20:36:02","2012-12-20 20:36:05","2012-12-20 20:36:07","1","247cff8e-5d8e-4f71-a92e-a2a690e83290",NULL,"ef936eeab9ef4111bf7b30c6f3bc653e","1be49e9b939a4cde9d829ffd36cc810c","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:36:02",NULL,"2012-12-20 20:36:05","Volume-410358",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:35:13","2012-12-20 20:35:17","2012-12-20 20:35:19","1","28a11cca-4a23-42b6-829c-43d80200a137",NULL,"e1c277843b04403e9fb481680fff1b61","222b561350db49f2a9333b9d58607506","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:35:13",NULL,"2012-12-20 20:35:17","Volume-988811",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:33:16","2012-12-20 20:33:20","2012-12-20 20:33:22","1","4a17d45b-7cb2-4d8c-9de0-081af58ef9d3",NULL,"3c6ebae5430649e48ee19e93c8c86709","f0056f65731b4c658243c545761bcbfb","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:33:17",NULL,"2012-12-20 20:33:20","Volume-88208",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:33:10","2012-12-20 20:33:13","2012-12-20 20:33:15","1","5d09466f-63ee-4383-a4d4-7d1eb343921b",NULL,"3c6ebae5430649e48ee19e93c8c86709","f0056f65731b4c658243c545761bcbfb","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:33:10",NULL,"2012-12-20 20:33:13","Volume-xml-731320",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:35:10","2012-12-20 20:35:10",NULL,"0","6cccd750-420c-4a99-b919-ef15d5db1734",NULL,"17536be8f6904d98a429009a945da02b","4fa7312c20914a8daa398769a9cf0cd8","blake","1","nova",NULL,NULL,NULL,"error","detached","2012-12-20 20:35:10",NULL,NULL,"Volume-89716",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:33:38","2012-12-20 20:33:39",NULL,"0","80150244-1fed-40e4-841a-e3fa534adfc7",NULL,"146bd66d5d584bce97885b92e4be6bf7","0e1335e202d74646b6cbb9af2cfb23e9","blake","1","nova",NULL,NULL,NULL,"error","detached","2012-12-20 20:33:38",NULL,NULL,"volume766408xml",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:33:03","2012-12-20 20:33:06","2012-12-20 20:33:08","1","85c2cf33-8775-47c6-9532-dbce334c8c3b",NULL,"248a2528e34b49fe973d62cc2c0dddde","2288c734f7e14ddab2532b2a61a6cc5e","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:33:03",NULL,"2012-12-20 20:33:06","Volume-852385",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:35:55","2012-12-20 20:35:59","2012-12-20 20:36:01","1","8ab73a2c-0265-4609-83e6-43c1de1a4d1b",NULL,"ef936eeab9ef4111bf7b30c6f3bc653e","1be49e9b939a4cde9d829ffd36cc810c","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:35:55",NULL,"2012-12-20 20:35:59","Volume-783892",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:03:48","2012-12-20 20:17:14","2012-12-20 20:17:17","1","92af9fc0-478f-45f7-9acc-788682d0d48e",NULL,"010785a6a3114b6492d54f3f3cfa39f0","d000979d31154b159966ba4eef7674b7","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:03:48",NULL,"2012-12-20 20:17:14",NULL,NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:36:12","2012-12-20 20:36:13",NULL,"0","9e755589-5a59-4df0-83d1-f9a2ec2b777b",NULL,"52eaaf423cb141379ee8b8c92512e71c","80f4477bbe07415796116618ab85ce1c","blake","1","nova",NULL,NULL,NULL,"error","detached","2012-12-20 20:36:13",NULL,NULL,"volume606406",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:32:56","2012-12-20 20:32:59","2012-12-20 20:33:01","1","ba3eab7f-2fb5-475f-81b8-02ae47781f17",NULL,"248a2528e34b49fe973d62cc2c0dddde","2288c734f7e14ddab2532b2a61a6cc5e","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:32:56",NULL,"2012-12-20 20:32:59","Volume-json-556422",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:33:24","2012-12-20 20:33:24",NULL,"0","bbc28867-5e23-4b15-8bc7-e768f84d9b8d",NULL,"cdf94dc40924446da1c6db19c7d6c32b","dc5533db4be04646825d0330e895ef41","blake","1","nova",NULL,NULL,NULL,"error","detached","2012-12-20 20:33:24",NULL,NULL,"volume-%s627208json",NULL,NULL,NULL,NULL,NULL
"2012-12-20 20:44:54","2012-12-20 20:45:00","2012-12-20 20:45:06","1","be2cbe7c-b6d5-4b7e-9d66-d484f6cf096a",NULL,"d5a29684f780441c8b793b6b87f99a55","50c17adc392b4df187dd47402eb97041","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:44:54",NULL,"2012-12-20 20:45:00","volume-414324",NULL,NULL,NULL,NULL,"ae75c976-27f1-4b34-80f0-a8ffea72e09a"
"2012-12-20 20:35:20","2012-12-20 20:35:23","2012-12-20 20:35:54","1","c5f2f83c-2665-4fa4-9f73-29b667009acd",NULL,"e1c277843b04403e9fb481680fff1b61","222b561350db49f2a9333b9d58607506","blake","1","nova",NULL,NULL,NULL,"deleted","detached","2012-12-20 20:35:20",NULL,"2012-12-20 20:35:23","Volume-313596",NULL,NULL,NULL,NULL,NULL

Clearly there is a clean up failure, which might be in tempest, or in cinder, or both.

[James E. Blair (corvus)]

The volume backing file in devstack-gate has been increased to 10G to work around this; when this is complete, that should be removed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/19320

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19320
Committed: http://github.com/openstack/tempest/commit/8ae54d6e9bf95eaaf651b7f57a08e51709896c5c
Submitter: Jenkins
Branch: master

commit 8ae54d6e9bf95eaaf651b7f57a08e51709896c5c
Author: john-griffith <email address hidden>
Date: Wed Jan 9 11:29:05 2013 -0700

    Add wait for resource deletion on volume teardown

    The teardown routines in some of the tests were not
    waiting for verification that the volume was actually deleted.
    As a result it's easy to intermittently hit a case where we run
    out of space for volume create in the following tests.

    This patch just adds a resource_deletion check/wait after
    each volume delete operation to make sure we don't get ahead
    of ourselves.

    Change-Id: I47f9ad2b8e6f79348c3d6188da3c24af0c6fe5d4
    Fixes: bug 1092689