Tempest test failure in ObjectTempUrlTest for Access denied

[Jadon Naas]

When running an automated test of OpenStack Ussuri on Ubuntu Focal, the Tempest test for ObjectTempUrlTest failed with an error of "Access denied". Here is the Tempest test results:

<testcase classname="tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest" id="f91c96d4-1230-4bba-8eb9-84476d18d991" name="test_get_object_using_temp_url" time="0.039" timestamp="2023-01-20T08:18:25+0000">
<failure>Traceback (most recent call last): File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/common/utils/__init__.py", line 89, in wrapper return func(*func_args, **func_kwargs) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/api/object_storage/test_object_temp_url.py", line 108, in test_get_object_using_temp_url resp, body = self.object_client.get(url) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 314, in get return self.request('GET', url, extra_headers, headers) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 720, in request self._error_checker(resp, resp_body) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 821, in _error_checker raise exceptions.Forbidden(resp_body, resp=resp) tempest.lib.exceptions.Forbidden: Forbidden Details: {'Code': 'AccessDenied', 'RequestId': 'tx0000046c9a5f4a6376561-0063ca4e51-294f-default', 'HostId': '294f-default-default'} </failure>
</testcase>
<testcase classname="tempest.api.object_storage.test_object_temp_url.ObjectTempUrlTest" id="9b08dade-3571-4152-8a4f-a4f2a873a735" name="test_put_object_using_temp_url" time="0.021" timestamp="2023-01-20T08:18:25+0000">
<failure>Traceback (most recent call last): File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/common/utils/__init__.py", line 89, in wrapper return func(*func_args, **func_kwargs) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/api/object_storage/test_object_temp_url.py", line 154, in test_put_object_using_temp_url resp, _ = self.object_client.put(url, new_data, None) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 363, in put return self.request('PUT', url, extra_headers, headers, body, chunked) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 720, in request self._error_checker(resp, resp_body) File "/home/ubuntu/snap/fcbtest/35/.rally/verification/verifier-560eb853-7f7a-4840-8269-2699b627d7a9/repo/tempest/lib/common/rest_client.py", line 821, in _error_checker raise exceptions.Forbidden(resp_body, resp=resp) tempest.lib.exceptions.Forbidden: Forbidden Details: {'Code': 'AccessDenied', 'RequestId': 'tx00000ffc5efd3b8f4a7c6-0063ca4e51-273a-default', 'HostId': '273a-default-default'} </failure>
</testcase>

There is not any apparent issue in the OpenStack logs, but the full crashdump and logs for the test can be found at https://oil-jenkins.canonical.com/artifacts/a598c233-aa49-4ffa-99f3-125f13e89f81/index.html.

[Tim Burke (1-tim-z)]

The link to more logs complains

> Either you have not been granted access to this resource or your entitlement has timed out. Please try again.

Is there any chance there was sizeable clock skew between client and server? I would've expected the default ~16min expiration time to generally be enough, though -- the only other explanation I can think of would be a mismatch in HMAC algorithm. Without more logs or proxy-server configs, it's hard to offer much more guidance.

[Felipe Reyes (freyes)]

According to [0], the storage object is provided by ceph-radosgw.

  ceph-radosgw:
    [...]
    channel: latest/stable
    charm: ceph-radosgw
    constraints: spaces=public-space
    num_units: 3
    options:
      region: RegionOne
      source: distro
    to:
    - lxd:0
    - lxd:1
    - lxd:2

Removing upstream swift and adding a task for ceph-radosgw.

[0] https://oil-jenkins.canonical.com/artifacts/a598c233-aa49-4ffa-99f3-125f13e89f81/generated/generated/openstack/bundle.yaml

[Peter Sabaini (peter-sabaini)]

Hey,

In the juju crashdump log there's a line that would hint at a VIP not yet setup, is it possible that this caused the fault?

hacluster-ceph-radosgw waiting 3 hacluster 2.0.3/stable 113 no Resource: res_cephrg_0e5c663_vip not yet configured

Is this a reproducible issue?

Thanks!

[Felipe Reyes (freyes)]

On Wed, 2023-05-24 at 14:39 +0000, Peter Sabaini wrote:
> Hey,
>
> In the juju crashdump log there's a line that would hint at a VIP not
> yet setup, is it possible that this caused the fault?
>
> hacluster-ceph-radosgw                       waiting          3
> hacluster                           2.0.3/stable   113  no
> Resource: res_cephrg_0e5c663_vip not yet configured
>
> Is this a reproducible issue?

yes, I've seen this issue in a perfectly healthy system. I believe the simplest way to reproduce it
would be to use charmed-openstack-tester with a modified bundle.

git clone https://github.com/openstack-charmers/charmed-openstack-tester.git
cd charmed-openstack-tester
# edit the bundle to drop swift and add ceph-radosgw - tests/distro-regression/tests/bundles/jammy-
antelope.yaml
tox -e func-target -- jammy-antelope