Rally cleanup servers raises exception

Bug #1584104 reported by Ivan Lozgachev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rally
Fix Released
High
Alexander Nevenchannyy

Bug Description

During execution of test boot_and_list_server.yaml Rally "failed" to cleanup created Nova instances. Actually, all instances have been deleted but Rally didn't understand that fact.

Environment (node roles):
1 Controller + Mongo
2 Compute
1 Cinder

Release:
MOS 9.0 build 308
Rally 28d2feba05817caa234f8d9391527a5e26d43942

Steps to reproduce:
Run attached boot_and_list_server.yaml scenario

Expected result:
Rally removed instances without errors

Actual result:
Rally removed instances and raised exception by mistake

Traced error:
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager [-] Instance 26cfbc00-f845-4c32-b2da-bc2ef1f3a83c could not be found. (HTTP 404) (Request-ID: req-03176e4a-d761-4903-88b0-9b94ed73b23d)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager Traceback (most recent call last):
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/manager.py", line 87, in _delete_single_resource
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager rutils.retry(resource._max_attempts, resource.delete)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/common/utils.py", line 208, in retry
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return func(*args, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/resources.py", line 111, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager super(NovaServer, self).delete()
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/rally/plugins/openstack/cleanup/base.py", line 123, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager self._manager().delete(self.id())
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/v2/servers.py", line 1303, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return self._delete("/servers/%s" % base.getid(server))
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/base.py", line 342, in _delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self.api.client.delete(url)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 467, in delete
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager return self._cs_request(url, 'DELETE', **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 436, in _cs_request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self._time_request(url, method, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 409, in _time_request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager resp, body = self.request(url, method, **kwargs)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager File "/root/rally/local/lib/python2.7/site-packages/novaclient/client.py", line 403, in request
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager raise exceptions.from_response(resp, body, url, method)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager NotFound: Instance 26cfbc00-f845-4c32-b2da-bc2ef1f3a83c could not be found. (HTTP 404) (Request-ID: req-03176e4a-d761-4903-88b0-9b94ed73b23d)
2016-05-20 14:40:53.614 17120 ERROR rally.plugins.openstack.cleanup.manager

Attaching Rally logs and scenario config

Revision history for this message
Ivan Lozgachev (ilozgachev) wrote :
Changed in rally:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Alexander Nevenchannyy (anevenchannyy) wrote :

This is bug also affects security_groups deletion.
Logs attached.

Revision history for this message
Alexander Nevenchannyy (anevenchannyy) wrote :
Changed in rally:
assignee: nobody → Alexander Nevenchannyy (anevenchannyy)
importance: Medium → High
status: Triaged → Fix Committed
Changed in rally:
milestone: none → 0.4.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to rally (master)

Reviewed: https://review.openstack.org/329989
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=6429e13617d977264135ee3d28068d8e8a1d73b5
Submitter: Jenkins
Branch: master

commit 6429e13617d977264135ee3d28068d8e8a1d73b5
Author: Alexandr Nevenchannyy <email address hidden>
Date: Wed Jun 15 17:41:47 2016 +0300

    Mark several cleanup resources as tenant_resource

    Nova servers and security groups are tenant related resources,
    but resource decorator missed that fact which makes cleanup
    tries to delete one resources several times.

    Change-Id: Ie3a0a40c2282fc7cffa33f1efe5d37601a461709
    Closes-Bug: #1584104

Changed in rally:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.