Swift capabilities tests fail when being run against Rados GW

[Pavlo Shchelokovskyy]

The swift api is organized as

host:port/info and host:port/v1/AUTH_<tenant-id>

with the latter being registered as Swift endpoint in the Keystone catalog.
Tempest tests for swift capabilities seem to cut all path from the API endpoint and append 'info' to it, however Ceph's RadosGW has a 'swift' prefix in its API (as it supports S3 api too) like

host:port/swift/info and host:port/swift/v1/AUTH_<tenant-id>

so those tests fail on RadosGW as they attempt to access the wrong capabilities url w/o swift prefix.

(also note that such tests will fail on Swift itself when it is deployed as vhost under Apache or other web server so the endpoint in catalog is like https://mycloud.com/object-store/v1/AUTH_<tenant-id> )

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/615558

[Attila Fazekas (afazekas)]

How other swift clients solves this ?
Does the swift documentation suggest anything about how get the info url from the catalog url ?

[Martin Kopec (mkopec)]

Adding example traceback as it's missing in the bug description (Stein deployment with RadosGW):

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 173, in setUpClass'
    b' six.reraise(etype, value, trace)'
    b' File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise'
    b' raise value'
    b' File "/usr/lib/python3.6/site-packages/tempest/test.py", line 166, in setUpClass'
    b' cls.resource_setup()'
    b' File "/usr/lib/python3.6/site-packages/tempest/api/object_storage/base.py", line 103, in resource_setup'
    b' api_prefix=CONF.object_storage.api_prefix)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/services/object_storage/capabilities_client.py", line 27, in list_capabilities'
    b' resp, body = self.get("/".join(filter(None, [api_prefix, \'info\'])))'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 298, in get'
    b" return self.request('GET', url, extra_headers, headers)"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 679, in request'
    b' self._error_checker(resp, resp_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 785, in _error_checker'
    b' raise exceptions.NotFound(resp_body, resp=resp)'
    b'tempest.lib.exceptions.NotFound: Object not found'
    b"Details: {'Code': 'NoSuchBucket', 'BucketName': 'info', 'RequestId': 'tx000000000000000021752-005d90a7bc-d3a1-default', 'HostId': 'd3a1-default-default'}"
    b''

I was testing the patch and I found that tempest.api.object_storage.test_healthcheck.HealthcheckTest.test_get_healthcheck needs to be edited too, because it's failing:
tempest.api.object_storage.test_healthcheck.HealthcheckTest.test_get_healthcheck [0.007523s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    b'Traceback (most recent call last):'
    b' File "/usr/lib/python3.6/site-packages/tempest/api/object_storage/test_healthcheck.py", line 31, in test_get_healthcheck'
    b' resp, _ = self.account_client.get("healthcheck", {})'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 298, in get'
    b" return self.request('GET', url, extra_headers, headers)"
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 679, in request'
    b' self._error_checker(resp, resp_body)'
    b' File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 770, in _error_checker'
    b' resp=resp)'
    b'tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided'
    b'Details: 404'
    b''

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    b'2019-09-29 12:51:47,335 481264 INFO [tempest.lib.common.rest_client] Request (HealthcheckTest:test_get_healthcheck): 404 GET http://10.0.0.125:8080/healthcheck 0.005s'
    b"2019-09-29 12:51:47,335 481264 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': '<omitted>'}"
    b' Body: None'
    b" Response - Headers: {'content-length': '221', ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/692375

[Ghanshyam Mann (ghanshyammann)]

This seems Tempest is not able to pickup the registered service catalog in keystone. In case of swift we are modifying it in a incorrect way.

For devstack it work well as registered swift endpoints does not have any extra path[1] and Tempest modify the base url fecthed from registered endpoints by truncating the whole path. If any cloud resigter swift endpoint with any extra path (like in case of Rados GW) then Tempest modified url is invalid.

So I agree to this issue now. I remember i was advocating this as invalid issue because of mis-reading this as modified API instead of customize service catalog.

For any other service's GET version API(most of them are with no path '\'), Temepst is doing correctly by making raw_request instead of going with rest_client filters way. Instead of passing things-to-do via rest client filters we should use the raw request way. Modifying the url via rest client filters way is very risky and not generic way. That always create the problem like this. In past we removed the _api_version from filters which was causing the similar problem of not supporting the customize endpoints and work only devstack-configured way.

[1] https://github.com/openstack/devstack/blob/bcb2c30c317834f62cab470dc0b58670d945a6ef/lib/swift#L646

[OpenStack Infra (hudson-openstack)]

Change abandoned by Oleksii Butenko (<email address hidden>) on branch: master
Review: https://review.opendev.org/615558
Reason: https://review.opendev.org/#/c/692375/6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692375
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=1d500e79156ada6bc6fdb628ed1da0efd4121f6a
Submitter: Zuul
Branch: master

commit 1d500e79156ada6bc6fdb628ed1da0efd4121f6a
Author: Martin Kopec <email address hidden>
Date: Thu Oct 31 13:56:42 2019 +0000

    Accept custom registered endpoints

    The review drops usage of skip_path() filter in the related tests
    and uses raw_request() instead.

    Normally a swift url is organised as host:port/info and
    host:port/v1/AUTH_<tenant-id>, see
    https://docs.openstack.org/api-ref/object-store/
    But RadosGW API is organised as host:port/swift/info and
    host:port/swift/v1/AUTH_<tenant-id>, see
    https://docs.ceph.com/docs/master/radosgw/config-ref/#swift-settings

    Close-bug: 1799981
    Change-Id: I6a932639a05defe0f04c600afcc35a19662937af

[Martin Kopec (mkopec)]

The fix has been merged.