Resource deletion failed

[Vlad Okhrimenko]

We get error when Rally tries to delete volumes

2016-03-18 07:42:13.767 11221 RALLYDEBUG rally.plugins.openstack.cleanup.manager [-] Cleaning up cinder transfers objects
2016-03-18 07:42:13.919 11221 RALLYDEBUG rally.plugins.openstack.cleanup.manager [-] Cleaning up cinder volumes objects
2016-03-18 07:42:14.107 11221 RALLYDEBUG rally.plugins.openstack.cleanup.manager [-] Deleting cinder volumes object a62774fe-7b1d-4136-a4e3-15b55764ada2
2016-03-18 07:42:14.390 11221 WARNING rally.plugins.openstack.cleanup.manager [-] Resource deletion failed, max retries exceeded for cinder.volumes: a62774fe-7b1d-4136-a4e3-15b55764ada2. Reaso
n: Invalid volume: Volume status must be available or error, but current status is: deleting. (HTTP 400) (Request-ID: req-0d058315-d575-4fc7-9d03-f10bed678ac1)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager [-] Invalid volume: Volume status must be available or error, but current status is: deleting. (HTTP 400) (Request-I
D: req-0d058315-d575-4fc7-9d03-f10bed678ac1)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager Traceback (most recent call last):
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/cleanup/manager.py", line 89, in _delete_sing
le_resource
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/common/utils.py", line 207, in retry
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/cleanup/base.py", line 117, in delete
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete(self.id())
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 289, in delete
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager return self._delete("/volumes/%s" % base.getid(volume))
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/cinderclient/base.py", line 309, in _delete
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager resp, body = self.api.client.delete(url)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 341, in delete
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager return self._cs_request(url, 'DELETE', **kwargs)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 295, in _cs_request
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager **kwargs)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 278, in request
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager raise exceptions.from_response(resp, body)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager BadRequest: Invalid volume: Volume status must be available or error, but current status is: deleting. (HTTP 400) (R
equest-ID: req-0d058315-d575-4fc7-9d03-f10bed678ac1)
2016-03-18 07:42:14.390 11221 ERROR rally.plugins.openstack.cleanup.manager
2016-03-18 07:42:14.392 11221 INFO rally.plugins.openstack.context.cleanup [-] Task 26f22390-234b-4508-977b-080a2a81a1e7 | Completed: user resources cleanup

[Chris St. Pierre (stpierre)]

Can you please paste the task configuration for a task where this occurs? Thanks.

[Vlad Okhrimenko (vokhrimenko)]

Unfortunately I don't have this data. But I think problem in another services (rabbit, cinder etc.), because volumes stay a 'deleting'-status. I suppose we need handler for this situation .

[Vlad Okhrimenko (vokhrimenko)]

sudo rally --log-file /var/log/rally.log --rally-debug task start /tmp/rally_tests/certification/openstack/task.yaml --task-args '{"service_list": ["authentication", "neutron", "cinder", "glance", "nova", "keystone"], "tenants_amount": "1", "image_name": "^(cirros.*uec|TestVM)$", "controllers_amount": "1", "glance_image_location": "", "computes_amount": "2", "use_existing_users": false, "flavor_name": "m1.tiny", "smoke": true, "users_amount": "1", "network_amount": "1", "storage_amount": "1"}'