Tempest cleanup not cleaning all leaked resources: not properly listing the resource

[Ghanshyam Mann]

Tempest cleanup command has multiple steps to cleanup the tempest created resources:

1. list the resources from the service side
2. filter those resources as per the tempest cleanup parameter. For example, if prefix is used then filter out the resources does not match the prefix.
3. delete the filtered resources.

There are some resources that were not cleaned up properly for example, Server Group
- tested in https://review.opendev.org/c/openstack/tempest/+/919783/5

Issue there was in step# 1 that we were not listing the server group properly and when it is fixed, it worked fine - https://review.opendev.org/c/openstack/tempest/+/897847/31..33/tempest/cmd/cleanup_service.py

Keypair also has issue that cleanup does not delete the leaked keypair - reproduced in https://review.opendev.org/c/openstack/tempest/+/919617/7

We should check if we are doing all the 3 steps properly or not. At least step#1 should be verified if issue we had for SG exist for other resources also.

[Lukas Piwowarski (lukas-piwowarski)]

The server group issue was fixed in this patch -> https://review.opendev.org/c/openstack/tempest/+/897847/

Regarding the keypair it seems like there is no simple way at the moment how to list ALL keypairs in a single "list keypairs" API call: "Lists keypairs that are associated with the account." [1]

We would have to probably iterate over the users created by tempest somehow and check whether the associated keypairs were deleted? Interesting read on this topic [2]

[1] https://docs.openstack.org/api-ref/compute/#list-keypairs
[2] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030854.html

[Martin Kopec (mkopec)]

I confirm that cleanup of server groups work as expected, I tested it this way:
1. I ran tempest with server groups cleanup commented out, similarly to Ghanshyam's patch https://review.opendev.org/c/openstack/tempest/+/919617
2. several server groups were left behind based on 'openstack server groups list --all-projects' output
3. I ran 'tempest cleanup --dry-run' and the generated dry_run.json contained the server groups
4. I ran 'tempest cleanup --dry-run --prefix tempest' and got the same result
5. I ran 'tempest cleanup --prefix <full name of one of a server group>' and the server group got successfully deleted
6. I reran steps 3 and 4 and the result obviously didn't contain the server group deleted in the step 5

I conclude that tempest cleanup can list and delete server groups.

Now, I'm gonna try the same with keypairs

[Martin Kopec (mkopec)]

btw, there is a bug in listing of keypairs if --prefix is passed

client.list_keypairs here: https://opendev.org/openstack/tempest/src/commit/6618aa253e04b8879ae6d721a48ee4851543ba4a/tempest/cmd/cleanup_service.py#L294

returns something like this:

[{'keypair': {'name': 'my_test', 'public_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL4FIBThf0PJRMrGFRYoItoGoUEoS7eZ15aBB2N5SB0G',
'fingerprint': '9d:21:43:71:d5:c1:32:7a:9c:25:95:17:42:5f:2a:e4'}}, {'keypair': {'name': 'my_test_2', 'public_key': 'ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIPrH86oiqlP+i/bCs5BxiwT6WI7k9wUEzDnFPoMtDSY9', 'fingerprint': '7a:51:78:fc:f9:dd:ce:da:e5:b9:db:6d:58:82:8d:d3'}}]

it's a list with dicts containing {'keypair': <details in a dict>} .. and therefore it fails couple lines below on this line: https://opendev.org/openstack/tempest/src/commit/6618aa253e04b8879ae6d721a48ee4851543ba4a/tempest/cmd/cleanup_service.py#L297

with:

Traceback (most recent call last):
  File "/home/ubuntu/tempest2/.tox/py310/lib/python3.10/site-packages/tempest/cmd/cleanup.py", line 155, in take_action
    self._cleanup()
  File "/home/ubuntu/tempest2/.tox/py310/lib/python3.10/site-packages/tempest/cmd/cleanup.py", line 255, in _cleanup
    svc.run()
  File "/home/ubuntu/tempest2/.tox/py310/lib/python3.10/site-packages/tempest/cmd/cleanup_service.py", line 150, in run
    self.dry_run()
  File "/home/ubuntu/tempest2/.tox/py310/lib/python3.10/site-packages/tempest/cmd/cleanup_service.py", line 323, in dry_run
    keypairs = self.list()
  File "/home/ubuntu/tempest2/.tox/py310/lib/python3.10/site-packages/tempest/cmd/cleanup_service.py", line 298, in list
    keypairs = self._filter_by_prefix(keypairs['keypair'])
TypeError: list indices must be integers or slices, not str

[Martin Kopec (mkopec)]

as stated in the comment #1, the listing keypairs is basically broken in tempest cleanup. Before we were able, as admin, to list all keypairs .. however, now, we can list only the keypairs that belong to the current user or we need to specify --user <user id> when listing keypairs of other users, we can't simply list them all

Technically, in tempest cleanup, we could iterate over all the users (before we delete them) and delete their keypairs, however, tempest cleanup is used to delete left over resources. In the situation when a keypair was left behind and the user got somehow deleted, we can't delete the keypair as we can't even list it (openstack keypair list --user <deleted user id> obviously fails with "No user with that name or ID ...". Well, in that case we can't even tell if keypairs were left behind as we can't list them.

The next course of action should be:
1. figure out what happens to a keypair if a user that the keypair belongs to is deleted - is the keypair deleted too? is it deleted by a garbage collector of sorts? etc
2. A. if 1. is true, then let's modify the cleanup to delete the kepairs in a loop as iterating over the leftover users and that's it, we don't have to worry about the keypairs of deleted users
2. B if 1. is not true, we need to figure out the official way of deleting keypairs without specific user id

[Martin Kopec (mkopec)]

After reading the email thread linked from the comment #1:
[2] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030854.html

it seems that the deletion of a keypair after deletion of the user associated with the keypair is not possible via openstack client .. it would be probably possible via a direct access to the DB, but that's not the point here, tempest does not do that.

It was concluded in the email, that deleting an openstack user before deleting their keypairs is a user error. Considering this and the fact that tempest does not access DB directly, there is only one thing we can do here from tempest cleanup perspective and that is delete user's keypairs before deleting the user:
https://opendev.org/openstack/tempest/src/commit/6618aa253e04b8879ae6d721a48ee4851543ba4a/tempest/cmd/cleanup_service.py#L1054

[Ghanshyam Mann (ghanshyammann)]

This is one of the open things we have in OpenStack: When any project or user is deleted, which is in Keystone, then any resources associated with that cannot be deleted from the service side. We have discussed it many times in many forums about deleting all the resources of any project or user when they are getting deleted, but there is no straight-forward solution for that.

> Before we were able, as admin, to list all keypairs ..

Is it? I am not sure if Nova is allowed to list all the keypairs even by admin. The user can list their own keypairs, or the admin can list the other user's kaypairs one by one, not all together. If we have changed anything on the Nova side regarding this, then it could be a straightforward backward-compatible change and done in microversion only.

As you mentioned, we need to iterate via the tempest created users and delete the associated keypair if any of them created by the tempest. But we need to do this in KeyPairService - https://opendev.org/openstack/tempest/src/commit/6618aa253e04b8879ae6d721a48ee4851543ba4a/tempest/cmd/cleanup_service.py#L287