tripleo

Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

Bug #1960355 reported by Douglas Viroel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo yoga-3

Bug Description

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) are failing in the followinf keystone tempest tests:
test_authorize_request_token
test_create_access_token
test_create_request_token
test_list_access_tokens
test_list_roles_for_access_token
test_revoke_access_token
test_show_role_for_access_token

with:
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/keystone_tempest_plugin/tests/api/identity/v3/test_oauth1_tokens.py", line 145, in test_authorize_request_token
    request_token = self._create_request_token(consumer)
  File "/usr/lib/python3.9/site-packages/keystone_tempest_plugin/tests/api/identity/v3/test_oauth1_tokens.py", line 41, in _create_request_token
    request_token = self.oauth_token_client.create_request_token(
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/identity/v3/oauth_token_client.py", line 130, in create_request_token
    resp, body = self.post(endpoint,
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 793, in _error_checker
    raise exceptions.UnexpectedContentType(str(resp.status),
tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided
Details: 500

https://logserver.rdoproject.org/56/36256/47/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/2a1da39/logs/undercloud/var/log/tempest/stestr_results.html.gz
https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/38f5ef7/logs/undercloud/var/log/tempest/stestr_results.html.gz

Revision history for this message
Arx Cruz (arxcruz) wrote :

Checking the logs, I notice this:

Response - Headers: {'date': 'Tue, 08 Feb 2022 19:00:30 GMT', 'server': 'Apache', 'content-length': '531', 'content-type': 'text/html; charset=iso-8859-1', 'connection': 'close', 'status': '500', 'content-location': 'https://[2001:db8:fd00:1000::5]:13000/v3/OS-OAUTH1/request_token'}
        Body: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>500 Internal Server Error</title>\n</head><body>\n<h1>Internal Server Error</h1>\n<p>The server encountered an internal error or\nmisconfiguration and was unable to complete\nyour request.</p>\n<p>Please contact the server administrator at \n [no address given] to inform them of the time this error occurred,\n and the actions you performed just before this error.</p>\n<p>More information about this error may be available\nin the server error log.</p>\n</body></html>\n'
2022-02-08 14:00:30,864 237273 INFO [tempest.lib.common.rest_client] Request (OAUTH1TokensTest:_run_cleanups): 204 DELETE https://[2001:db8:fd00:1000::5]:13000/v3/OS-OAUTH1/consumers/aad136f7d88642808a01baf2e3cd3ef5 0.081s
2022-02-08 14:00:30,865 237273 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}

Probably it's a bug in keystone?

Revision history for this message
Arx Cruz (arxcruz) wrote :

Also, it's being hard to find the proper request, but in the timestamp, I got this:

2022-02-08 19:00:30.293 117 DEBUG keystone.auth.core [req-8b5a104c-9bb6-463a-9e65-b8a2e1e7ba45 - - - - -] MFA Rules not processed for user `6f72adab6a0f4209935f9f2c53c64bef`. Rule list: `[]` (Enabled: `True`). check_auth_methods_against_rules /usr/lib/python3.9/site-packages/keystone/auth/core.py:438

And checking the code, this throws an exception InsufficientAuthMethods:

https://opendev.org/openstack/keystone/src/branch/master/keystone/api/_shared/authentication.py#L217

Revision history for this message
Douglas Viroel (dviroel) wrote :

Last successful job [1] was still using 'admin' endpoint url. Failing ones use 'public' endpoint url[2].

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/8a14d6c/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz
[2] https://logserver.rdoproject.org/56/36256/47/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/2a1da39/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz

Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

Hello team,

We are hitting the same issue on 17 on rhel-9 in fs035 job as well.

Revision history for this message
Bhagyashri Shewale (bhagyashri-shewale) wrote :

tempest skiplist patch https://review.opendev.org/q/4ecd42032b128fde0a9b87427af0d2b8725b2978

Revision history for this message
Jakob Meng (jm1337) wrote :

@dasm's skiplist revert patch:

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/831185

@dasm's testproject for skiplist revert patch:

https://review.rdoproject.org/r/c/testproject/+/39955

As of 2022-05-18 the testproject is still failing on jobs

  periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master
  periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

for all tempest test cases in test group

  keystone_tempest_plugin.tests.api.identity.v3.test_oauth1_tokens.OAUTH1TokensTest

https://logserver.rdoproject.org/55/39955/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/a8181ec/logs/undercloud/var/log/tempest/stestr_results.html.gz

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote (last edit ):

Recent test run shows some details wrt source of failure [1] [2]

> ValueError: invalid literal for int() with base 10: 'db8:fd00:1000::5]:13000'
...
> [Wed May 18 20:56:45.061930 2022] [wsgi:error] [pid 136:tid 147] [remote fd00:fd00:fd00:2000::2f7:49336] File "/usr/lib/python3.9/site-packages/oauthlib/oauth1/rfc5849/signature.py", line 187, in base_string_uri
[Wed May 18 20:56:45.061932 2022] [wsgi:error] [pid 136:tid 147] [remote fd00:fd00:fd00:2000::2f7:49336] raise ValueError('port is not an integer')
[Wed May 18 20:56:45.061939 2022] [wsgi:error] [pid 136:tid 147] [remote fd00:fd00:fd00:2000::2f7:49336] ValueError: port is not an integer

[1]: https://logserver.rdoproject.org/55/39955/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/a8181ec/logs/overcloud-controller-2/var/log/containers/httpd/keystone/keystone_wsgi_error.log.txt.gz
[2]: https://logserver.rdoproject.org/55/39955/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/799f8df/logs/overcloud-controller-1/var/log/containers/httpd/keystone/keystone_wsgi_error.log.txt.gz

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote (last edit ):

The issue seems to be connected to the way how oauthlib is getting port number[1] by splitting IP using ":" [2].
With IPv6 it's causing issues.
Our tests use oauthlib 3.1.1 [3]

[1]: https://github.com/oauthlib/oauthlib/blob/master/oauthlib/oauth1/rfc5849/signature.py#L185
[2]: https://github.com/oauthlib/oauthlib/blob/master/oauthlib/oauth1/rfc5849/signature.py#L176
[3]: https://logserver.rdoproject.org/55/39955/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/799f8df/logs/overcloud-controller-1/var/log/extra/pip.txt.gz

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote (last edit ):

I confirmed the problem with an oauthlib

> >>> import oauthlib
>>> oauthlib.__version__
'3.2.0'
>>> from oauthlib import oauth1
>>> oauth1.rfc5849.signature.base_string_uri("https://[123:db8:fd00:1000::5]:13000")
Traceback (most recent call last):
  File "/home/dasm/git/venv/lib64/python3.10/site-packages/oauthlib/oauth1/rfc5849/signature.py", line 185, in base_string_uri
    port_num = int(port_str) # try to parse into an integer number
ValueError: invalid literal for int() with base 10: 'db8:fd00:1000::5]:13000'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dasm/git/venv/lib64/python3.10/site-packages/oauthlib/oauth1/rfc5849/signature.py", line 187, in base_string_uri
    raise ValueError('port is not an integer')
ValueError: port is not an integer

I submitted upstream bug: https://github.com/oauthlib/oauthlib/issues/817
My attempt to fix the issue can be found here: https://github.com/oauthlib/oauthlib/pull/818

Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :

Upstream pull request got merged few minutes ago. I'm gonna keep this bug report open, until we can verify it's fixed.

Ronelle Landy (rlandy)
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.