keystonemiddleware

Unit test failures with python 3.10+ "Max retries exceeded with url: /v3/ec2tokens"

Bug #2023689 reported by Corey Bryant on 2023-06-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	keystonemiddleware	In Progress	Undecided	Douglas Mendizábal

Bug Description

These tests fail on python 3.10 and above and perhaps on prior versions of python that I haven't tested.

lxc launch ubuntu-daily:mantic m1
lxc exec m1 /bin/bash
root@m1:~# git clone https://opendev.org/openstack/keystonemiddleware
root@m1:~# cd keystonemiddleware
root@m1:~/keystonemiddleware# python3 --version
Python 3.11.4
root@m1:~/keystonemiddleware# tox -e py311
...
==============================
Failed 4 tests - output below:
==============================

keystonemiddleware.tests.unit.test_ec2_token_middleware.EC2TokenMiddlewareTestBad.test_communication_failure
------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/eventlet/greenio/base.py", line 270, in connect
socket_checkerr(fd)

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/eventlet/greenio/base.py", line 54, in socket_checkerr
raise socket.error(err, errno.errorcode[err])

ConnectionRefusedError: [Errno 111] ECONNREFUSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)

File "/usr/lib/python3.11/http/client.py", line 1286, in request
self._send_request(method, url, body, headers, encode_chunked)

File "/usr/lib/python3.11/http/client.py", line 1332, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)

File "/usr/lib/python3.11/http/client.py", line 1281, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)

File "/usr/lib/python3.11/http/client.py", line 1041, in _send_output
self.send(msg)

File "/usr/lib/python3.11/http/client.py", line 979, in send
self.connect()

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(

urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fcd7b32e910>: Failed to establish a new connection: [Errno 111] ECONNREFUSED

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v3/ec2tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcd7b32e910>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

      File "/usr/lib/python3.11/unittest/mock.py", line 1375, in patched
    return func(*newargs, **newkeywargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/keystonemiddleware/tests/unit/test_ec2_token_middleware.py", line 164, in test_communication_failure
    resp = req.get_response(self.middleware)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/webob/request.py", line 1313, in send
    status, headers, app_iter = self.call_application(
                                ^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/webob/request.py", line 1278, in call_application
    app_iter = application(self.environ, start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/keystonemiddleware/ec2_token.py", line 177, in __call__
    response = requests.post(CONF.keystone_ec2_token.url,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

      File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/keystonemiddleware/.tox/py311/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
raise ConnectionError(e, request=request)

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /v3/ec2tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcd7b32e910>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
Starting the EC2 Token Authentication component
Starting new HTTP connection (1): localhost:5000

keystonemiddleware.tests.unit.test_ec2_token_middleware.EC2TokenMiddlewareTestGood.test_protocol_old_versions
-------------------------------------------------------------------------------------------------------------