Tempest cleanup not cleaning all leaked resources: not properly listing the resource

[Ghanshyam Mann]

Tempest cleanup command has multiple steps to cleanup the tempest created resources:

1. list the resources from the service side
2. filter those resources as per the tempest cleanup parameter. For example, if prefix is used then filter out the resources does not match the prefix.
3. delete the filtered resources.

There are some resources that were not cleaned up properly for example, Server Group
- tested in https://review.opendev.org/c/openstack/tempest/+/919783/5

Issue there was in step# 1 that we were not listing the server group properly and when it is fixed, it worked fine - https://review.opendev.org/c/openstack/tempest/+/897847/31..33/tempest/cmd/cleanup_service.py

Keypair also has issue that cleanup does not delete the leaked keypair - reproduced in https://review.opendev.org/c/openstack/tempest/+/919617/7

We should check if we are doing all the 3 steps properly or not. At least step#1 should be verified if issue we had for SG exist for other resources also.

[Lukas Piwowarski (lukas-piwowarski)]

The server group issue was fixed in this patch -> https://review.opendev.org/c/openstack/tempest/+/897847/

Regarding the keypair it seems like there is no simple way at the moment how to list ALL keypairs in a single "list keypairs" API call: "Lists keypairs that are associated with the account." [1]

We would have to probably iterate over the users created by tempest somehow and check whether the associated keypairs were deleted? Interesting read on this topic [2]

[1] https://docs.openstack.org/api-ref/compute/#list-keypairs
[2] https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030854.html