OpenStack Shared File Systems Service (Manila)

[tempest] Openstack resources are not deleted post tempest cleanup

Bug #1830943 reported by Vida Haririan on 2019-05-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Invalid	Undecided	Martin Kopec
	tempest	Fix Released	Medium	Martin Kopec

Bug Description

Tempest cleanup utility does not always delete and destroy used OpenStack resources.

**Reproduce Steps:
$ tempest cleanup --init-saved-state
$ execute selected manila upstream api tests
$ tempest cleanup

**User impact: Tests may fail due to insufficient resources.

**Failure example:
setUpClass (manila_tempest_tests.tests.api.admin.test_user_messages.UserMessageTest)
------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 173, in setUpClass'
    b' six.reraise(etype, value, trace)'
    b' File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise'
    b' raise value'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 158, in setUpClass'
    b' cls.setup_credentials()'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 377, in setup_credentials'
    b' credential_type=credentials_type)'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 682, in get_client_manager'
    b' creds = getattr(cred_provider, credentials_method)()'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 356, in get_admin_creds'
    b" return self.get_credentials('admin')"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 345, in get_credentials'
    b' credentials.tenant_id)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 262, in _create_network_resources'
    b' router = self._create_router(router_name, tenant_id)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/dynamic_creds.py", line 323, in _create_router'
    b' resp_body = self.routers_admin_client.create_router(**kwargs)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/routers_client.py", line 27, in create_router'
    b' return self.create_resource(uri, post_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 61, in create_resource'
    b' resp, body = self.post(req_uri, req_post_data)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post'
    b" return self.request('POST', url, extra_headers, headers, body, chunked)"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 679, in request'
    b' self._error_checker(resp, resp_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 800, in _error_checker'
    b' raise exceptions.Conflict(resp_body, resp=resp)'
    b'tempest.lib.exceptions.Conflict: Conflict with state of target resource'
    b"Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network 166b710f-cb4a-4eaa-af74-269964078dbe.', 'detail': ''}"
    b''

**Reference: tempest.conf & saved_state.json files are attached

**Screen captures:
(overcloud) [stack@undercloud-0 testing_dir]$ ll saved_state.json
-rw-rw-r--. 1 stack stack 2675 May 23 17:56 saved_state.json

(overcloud) [stack@undercloud-0 testing_dir]$ tempest cleanup
Begin cleanup
Process 0 projects

Tags:

Revision history for this message

Vida Haririan (vhariria) wrote on 2019-05-29:

saved_state.json Edit (2.6 KiB, application/json)

Revision history for this message

Vida Haririan (vhariria) wrote on 2019-05-29:

tempest.conf Edit (6.8 KiB, text/plain)

Jason Grosso (jgrosso) on 2019-06-11

Changed in manila:
assignee:	nobody → Martin Kopec (mkopec)

Revision history for this message

Martin Kopec (mkopec) wrote on 2019-06-12:

Tempest cleanup works with two types of services, project ones [1] and global ones [2]. First cleanup iterates over the projects which are not in saved_state.json and removes all resources related to that projects. Then it iterates over the global services and removes them.

If a resource is associated with a project which has been already removed and the resource is from project services category, it won't be cleaned.
I tried to run some of the manila tests and after cleanup I ended up with 50 networks which were associated with already non-existing projects and therefore they weren't cleaned. I guess this could happen if a test case removes a project but doesn't remove the resources associated with the project first. I'm gonna have a look how it can be solved.

[1] https://opendev.org/openstack/tempest/src/commit/77febec7848d2d031954619b6ce442c9723d0043/tempest/cmd/cleanup_service.py#L961
[2] https://opendev.org/openstack/tempest/src/commit/77febec7848d2d031954619b6ce442c9723d0043/tempest/cmd/cleanup_service.py#L988

Revision history for this message

Vida Haririan (vhariria) wrote on 2019-09-05:

Hi Martin, Have you had a chance to look at this issue or have an update to share?

Revision history for this message

Martin Kopec (mkopec) wrote on 2019-09-20:

Hi vhariria,

I'm sorry for my late response. In the previous comment I explained how tempest cleanup currently works. In order to remove these resources we'll need to add a new logic to cleanup which will somehow look for resources without a project/with project which is already deleted.

I'm wondering why I was able to reproduce the issue mainly after I ran manila plugin. Could you please have a look at the tests and check if every test which creates resources also deletes them? I would also check if the resources are deleted in the correct order, because I can't otherwise explain how I ended up with many resources left belonging to non-existing projects (mainly networks).
One part of the solution should be prevention, so please, check the cleanup of the tests.

The other part of the solution is to add the logic, deleting "zombie" resources, to tempest cleanup. I'll check meanwhile what are our options regarding that.

Revision history for this message

Vida Haririan (vhariria) wrote on 2019-09-20:

Hi mkopec,

Thanks for your feedback. Yes, a cleanup routine is called at end of each Manila test, adding lkuchlani, who manages the routines, for further feedback.

Revision history for this message

Martin Kopec (mkopec) wrote on 2019-09-21:

I have a workaround, until I get it implemented and merged in tempest, you can do the following:
1. Create a simple project, f.e.: $ openstack project create test
2. Run tempest cleanup

The project you create will make tempest cleanup iterate over the resources and then they will be properly cleaned.

Revision history for this message

Martin Kopec (mkopec) wrote on 2019-10-10:

It's interesting that the workaround I mentioned in the previous comment is working - it means that the current workflow of cleaning resources is not accurate.

The current workflow is that the tool iterates over projects and deletes all resources tied to a particular project. However, as the workaround proofs, some services can be deleted all at once without iteration over the projects.

Such a new workflow would be not only faster (because the tool wouldn't have to list all resources for every project, it could simply list all the resources once) but it also fixes the issue (listing resources wouldn't be tied to a project thus the resources will be listed and handled accordingly even in case there are no other projects apart from those in saved_state.json).
In order to implement it I first need to get fixed the following bug because it simplifies a few things.
https://launchpad.net/bugs/1766582

Revision history for this message

Martin Kopec (mkopec) wrote on 2019-11-13:

A patch fixing this has been proposed: https://review.opendev.org/#/c/694064/

Revision history for this message

Ghanshyam Mann (ghanshyammann) wrote on 2019-12-20:

#10

yeah, we do not need to iterate via projects for resources not associated with projects. All those resources can be deleted directly.

Changed in manila:
status:	New → Invalid
Changed in tempest:
status:	New → Triaged
importance:	Undecided → Medium
assignee:	nobody → Martin Kopec (mkopec)

OpenStack Infra (hudson-openstack) on 2019-12-20

Changed in tempest:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-01-02: Fix merged to tempest (master)

#11

Reviewed: https://review.opendev.org/694064
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=56c0b2be78faea800af73596e868bdf0969b0544
Submitter: Zuul
Branch: master

commit 56c0b2be78faea800af73596e868bdf0969b0544
Author: Martin Kopec <email address hidden>
Date: Wed Nov 13 12:50:07 2019 +0000

Tempest cleanup: improve iterating over projects

    Currently cleanup iterates over projects and deletes all resources
    tied to a particular project. However, most of the resources can be
    deleted all at once without iteration over the projects.
    This will fix the mentioned bug and also make the tool much more
    efficient as we'll avoid many iterations and queries.

Closes-bug: #1830943
Change-Id: I82c7899a32bee0e714e71d703195085e10ea224f