[tempest] Openstack resources are not deleted post tempest cleanup

Bug #1830943 reported by V on 2019-05-29
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Manila
Undecided
Martin Kopec

Bug Description

Tempest cleanup utility does not always delete and destroy used OpenStack resources.

**Reproduce Steps:
$ tempest cleanup --init-saved-state
$ execute selected manila upstream api tests
$ tempest cleanup

**User impact: Tests may fail due to insufficient resources.

**Failure example:
setUpClass (manila_tempest_tests.tests.api.admin.test_user_messages.UserMessageTest)
------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 173, in setUpClass'
    b' six.reraise(etype, value, trace)'
    b' File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise'
    b' raise value'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 158, in setUpClass'
    b' cls.setup_credentials()'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 377, in setup_credentials'
    b' credential_type=credentials_type)'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 682, in get_client_manager'
    b' creds = getattr(cred_provider, credentials_method)()'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 356, in get_admin_creds'
    b" return self.get_credentials('admin')"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 345, in get_credentials'
    b' credentials.tenant_id)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 262, in _create_network_resources'
    b' router = self._create_router(router_name, tenant_id)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 323, in _create_router'
    b' resp_body = self.routers_admin_client.create_router(**kwargs)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/routers_client.py", line 27, in create_router'
    b' return self.create_resource(uri, post_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 61, in create_resource'
    b' resp, body = self.post(req_uri, req_post_data)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post'
    b" return self.request('POST', url, extra_headers, headers, body, chunked)"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 679, in request'
    b' self._error_checker(resp, resp_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 800, in _error_checker'
    b' raise exceptions.Conflict(resp_body, resp=resp)'
    b'tempest.lib.exceptions.Conflict: Conflict with state of target resource'
    b"Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network 166b710f-cb4a-4eaa-af74-269964078dbe.', 'detail': ''}"
    b''

**Reference: tempest.conf & saved_state.json files are attached

**Screen captures:
(overcloud) [stack@undercloud-0 testing_dir]$ ll saved_state.json
-rw-rw-r--. 1 stack stack 2675 May 23 17:56 saved_state.json

(overcloud) [stack@undercloud-0 testing_dir]$ openstack network list
+--------------------------------------+------------------------------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+------------------------------------------+--------------------------------------+
| 03e522a1-c3bd-421f-84ce-30bbf164dbd9 | tempest-share-service-1723054483-network | a0b7670c-b605-4504-9b4a-7e7f1af2032e |
| 0b53019a-0bcd-4351-8df0-6eb0fed3793e | tempest-share-service-1820279606-network | c63bf07c-8a27-4516-aa07-da24377912cd |
| 0fab7ada-920f-4847-87bf-40fd7b2d7882 | tempest-share-service-1322603070-network | a2529d4f-c6f1-46af-b73c-39d9d6dfd971 |
| 110d202f-e49a-4a81-87a8-92e88698c75e | tempest-share-service-1051501383-network | ebcf2d0f-30c4-4a84-8463-dace40089e20 |
| 166b710f-cb4a-4eaa-af74-269964078dbe | public | d4e8879e-a11e-4a42-83b3-1fdccad633ae |
| 249d773b-f555-492d-a88c-8b59a5db0ac5 | tempest-share-service-1237827927-network | d40f3765-5fdd-4313-b991-0a9fbfdba00b |
...

(overcloud) [stack@undercloud-0 testing_dir]$ tempest cleanup
Begin cleanup
Process 0 projects

(overcloud) [stack@undercloud-0 testing_dir]$ openstack network list
+--------------------------------------+------------------------------------------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+------------------------------------------+--------------------------------------+
| 03e522a1-c3bd-421f-84ce-30bbf164dbd9 | tempest-share-service-1723054483-network | a0b7670c-b605-4504-9b4a-7e7f1af2032e |
| 0b53019a-0bcd-4351-8df0-6eb0fed3793e | tempest-share-service-1820279606-network | c63bf07c-8a27-4516-aa07-da24377912cd |
| 0fab7ada-920f-4847-87bf-40fd7b2d7882 | tempest-share-service-1322603070-network | a2529d4f-c6f1-46af-b73c-39d9d6dfd971 |
| 110d202f-e49a-4a81-87a8-92e88698c75e | tempest-share-service-1051501383-network | ebcf2d0f-30c4-4a84-8463-dace40089e20 |
| 166b710f-cb4a-4eaa-af74-269964078dbe | public | d4e8879e-a11e-4a42-83b3-1fdccad633ae |
| 249d773b-f555-492d-a88c-8b59a5db0ac5 | tempest-share-service-1237827927-network | d40f3765-5fdd-4313-b991-0a9fbfdba00b |
...

V (vhariria) wrote :
V (vhariria) wrote :
Jason Grosso (jgrosso) on 2019-06-11
Changed in manila:
assignee: nobody → Martin Kopec (mkopec)
Martin Kopec (mkopec) wrote :

Tempest cleanup works with two types of services, project ones [1] and global ones [2]. First cleanup iterates over the projects which are not in saved_state.json and removes all resources related to that projects. Then it iterates over the global services and removes them.

If a resource is associated with a project which has been already removed and the resource is from project services category, it won't be cleaned.
I tried to run some of the manila tests and after cleanup I ended up with 50 networks which were associated with already non-existing projects and therefore they weren't cleaned. I guess this could happen if a test case removes a project but doesn't remove the resources associated with the project first. I'm gonna have a look how it can be solved.

[1] https://opendev.org/openstack/tempest/src/commit/77febec7848d2d031954619b6ce442c9723d0043/tempest/cmd/cleanup_service.py#L961
[2] https://opendev.org/openstack/tempest/src/commit/77febec7848d2d031954619b6ce442c9723d0043/tempest/cmd/cleanup_service.py#L988

V (vhariria) wrote :

Hi Martin, Have you had a chance to look at this issue or have an update to share?

Martin Kopec (mkopec) wrote :

Hi vhariria,

I'm sorry for my late response. In the previous comment I explained how tempest cleanup currently works. In order to remove these resources we'll need to add a new logic to cleanup which will somehow look for resources without a project/with project which is already deleted.

I'm wondering why I was able to reproduce the issue mainly after I ran manila plugin. Could you please have a look at the tests and check if every test which creates resources also deletes them? I would also check if the resources are deleted in the correct order, because I can't otherwise explain how I ended up with many resources left belonging to non-existing projects (mainly networks).
One part of the solution should be prevention, so please, check the cleanup of the tests.

The other part of the solution is to add the logic, deleting "zombie" resources, to tempest cleanup. I'll check meanwhile what are our options regarding that.

V (vhariria) wrote :

Hi mkopec,

Thanks for your feedback. Yes, a cleanup routine is called at end of each Manila test, adding lkuchlani, who manages the routines, for further feedback.

Martin Kopec (mkopec) wrote :

I have a workaround, until I get it implemented and merged in tempest, you can do the following:
1. Create a simple project, f.e.: $ openstack project create test
2. Run tempest cleanup

The project you create will make tempest cleanup iterate over the resources and then they will be properly cleaned.

Martin Kopec (mkopec) wrote :

It's interesting that the workaround I mentioned in the previous comment is working - it means that the current workflow of cleaning resources is not accurate.

The current workflow is that the tool iterates over projects and deletes all resources tied to a particular project. However, as the workaround proofs, some services can be deleted all at once without iteration over the projects.

Such a new workflow would be not only faster (because the tool wouldn't have to list all resources for every project, it could simply list all the resources once) but it also fixes the issue (listing resources wouldn't be tied to a project thus the resources will be listed and handled accordingly even in case there are no other projects apart from those in saved_state.json).
In order to implement it I first need to get fixed the following bug because it simplifies a few things.
https://launchpad.net/bugs/1766582

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers