tempest cleanup NotImplemented error

[Martin Kopec]

We hit NotImplemented error during initializing a saved state:
019-05-29 17:46:30 | + /usr/bin/tempest cleanup --init-saved-state
2019-05-29 17:46:38 | Traceback (most recent call last):
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 98, in take_action
2019-05-29 17:46:38 | self.init(parsed_args)
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 126, in init
2019-05-29 17:46:38 | self._init_state()
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup.py", line 309, in _init_state
2019-05-29 17:46:38 | svc.run()
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 132, in run
2019-05-29 17:46:38 | self.save_state()
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 705, in save_state
2019-05-29 17:46:38 | pools = self.list()
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/cmd/cleanup_service.py", line 680, in list
2019-05-29 17:46:38 | pools = client.list_subnetpools(**self.tenant_filter)['subnetpools']
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/subnetpools_client.py", line 28, in list_subnetpools
2019-05-29 17:46:38 | return self.list_resources(uri, **filters)
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/services/network/base.py", line 34, in list_resources
2019-05-29 17:46:38 | resp, body = self.get(req_uri)
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 294, in get
2019-05-29 17:46:38 | return self.request('GET', url, extra_headers, headers)
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
2019-05-29 17:46:38 | self._error_checker(resp, resp_body)
2019-05-29 17:46:38 | File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 842, in _error_checker
2019-05-29 17:46:38 | message=message)
2019-05-29 17:46:38 | NotImplemented: Got NotImplemented error

We got the error when tempest cleanup wanted to list subnet pools available [1].

In order to assume that subnet pools api is available, subnet_allocation extension is expected to be present [2] in network-feature-enabled.api_extensions list in tempest.conf. Our tempest.conf didn't contain such extension.

Tempest cleanup doesn't check if the apis it calls in order to clean resources are enabled in the tempest.conf.
In my opinion it doesn't have to check explicitly the extension enabled, it would be enough if tempest cleanup caught NotImplemented exception, logged it and continued. If an api is not implemented, then there are no resources to clean so it can safely continue.

[1] https://opendev.org/openstack/tempest/src/commit/e681998023d9714da346d117e18ab673b5657f71/tempest/cmd/cleanup_service.py#L681
[2] https://docs.openstack.org/newton/networking-guide/config-subnet-pools.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/664962

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664962
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=97857948d9cdd7cec21e2542fa2222e820e5a66a
Submitter: Zuul
Branch: master

commit 97857948d9cdd7cec21e2542fa2222e820e5a66a
Author: Martin Kopec <email address hidden>
Date: Wed Jun 12 15:23:21 2019 +0000

    Fix cleanup NotImplemented error

    tempest cleanup doesn't check if the APIs it uses are
    implemented or not. Therefore the patch adds a try
    except block preventing tempest cleanup to fail when
    some of the APIs are not implemented.

    Closes-bug: #1832566

    Change-Id: I9ced4af40eb0c2a22e3557caded56045a397d539

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 21.0.0 release.

[Martin Kopec (mkopec)]

The fix seems to be not complete because now when we run 'tempest cleanup --init-saved-state' and we have for example the subnet pool disabled, like the bug description mentioned, it will return a non-zero return code.
it should return zero return code, because the resources were discovered correctly.

From automation perspective if the subnetpools are disabled in tempest.conf, which is a valid case, the script will generate proper saved_state.json file but will exit with code which is interpreted as an errror.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/674689

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/674689
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=219e322e7eced4b4d7a3a8930f62bf64baf05cd9
Submitter: Zuul
Branch: master

commit 219e322e7eced4b4d7a3a8930f62bf64baf05cd9
Author: Martin Kopec <email address hidden>
Date: Mon Aug 5 20:02:20 2019 +0000

    tempest cleanup fix return code

    It's a follow up fix for tempest cleanup NotImplemented error issue.
    Tempest cleanup shouldn't return a non-zero return code for
    NotImplemented errors as they are actually not errors, they are just
    an outcome of tempest not checking availability of extensions in order
    to keep it simple without any complicated logic.

    Closes-bug: #1832566

    Change-Id: I09e28c245c11dd3dd061df786bd137eceb97be07

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 22.0.0 release.