Rally cleanup servers raises exception

Bug #1584104 reported by Ivan Lozgachev on 2016-05-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rally
High
Alexander Nevenchannyy

Bug Description

During execution of test boot_and_list_server.yaml Rally "failed" to cleanup created Nova instances. Actually, all instances have been deleted but Rally didn't understand that fact.

Environment (node roles):
1 Controller + Mongo
2 Compute
1 Cinder

Release:
MOS 9.0 build 308
Rally 28d2feba05817caa234f8d9391527a5e26d43942

Steps to reproduce:
Run attached boot_and_list_server.yaml scenario

Expected result:
Rally removed instances without errors

Actual result:
Rally removed instances and raised exception by mistake

Traced error:
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager [-] Instance 26cfbc00-f845-4c32-b2da-bc2ef1f3a83c could not be found. (HTTP 404) (Request-ID: req-03176e4a-d761-4903-88b0-9b94ed73b23d)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager Traceback (most recent call last):
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/manager.py", line 87, in _delete_single_resource
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/common/utils.py", line 208, in retry
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/resources.py", line 111, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager super(NovaServer, self).delete()
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/base.py", line 123, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete(self.id())
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1303, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return self._delete("/servers/%s" % base.getid(server))
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/base.py", line 342, in _delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self.api.client.delete(url)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 467, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return self._cs_request(url, 'DELETE', **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 436, in _cs_request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self._time_request(url, method, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 409, in _time_request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self.request(url, method, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 403, in request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager raise exceptions.from_response(resp, body, url, method)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager NotFound: Instance 26cfbc00-f845-4c32-b2da-bc2ef1f3a83c could not be found. (HTTP 404) (Request-ID: req-03176e4a-d761-4903-88b0-9b94ed73b23d)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager

Attaching Rally logs and scenario config

Ivan Lozgachev (ilozgachev) wrote :
Changed in rally:
status: New → Triaged
importance: Undecided → Medium

This is bug also affects security_groups deletion.
Logs attached.

Changed in rally:
assignee: nobody → Alexander Nevenchannyy (anevenchannyy)
importance: Medium → High
status: Triaged → Fix Committed
Changed in rally:
milestone: none → 0.4.1

Reviewed: https://review.openstack.org/329989
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=6429e13617d977264135ee3d28068d8e8a1d73b5
Submitter: Jenkins
Branch: master

commit 6429e13617d977264135ee3d28068d8e8a1d73b5
Author: Alexandr Nevenchannyy <email address hidden>
Date: Wed Jun 15 17:41:47 2016 +0300

    Mark several cleanup resources as tenant_resource

    Nova servers and security groups are tenant related resources,
    but resource decorator missed that fact which makes cleanup
    tries to delete one resources several times.

    Change-Id: Ie3a0a40c2282fc7cffa33f1efe5d37601a461709
    Closes-Bug: #1584104

Changed in rally:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers