on octavia-dasboard master:
the dashboard cannot get the list of loadbalancers:
horizon_error.log:
2023-12-17 20:25:36.579895 DEBUG keystoneauth.session REQ: curl -g -i -X GET http://192.168.1.101/identity -H "Accept: application/json" -H "User-Agent: openstacksdk/2.0.1 keystoneauth1/5.4.0 python-requests/2.31.0 CPython/3.9.18"
2023-12-17 20:25:36.585039 DEBUG keystoneauth.session RESP: [300] Connection: close Content-Length: 272 Content-Type: application/json Date: Sun, 17 Dec 2023 20:25:36 GMT Location: http://192.168.1.101/identity/v3/ Server: Apache/2.4.57 (CentOS Stream) OpenSSL/3.0.7 mod_wsgi/4.7.1 Python/3.9 Vary: X-Auth-Token x-openstack-request-id: req-bc1f979a-d01a-4e7a-b486-bc446860215d
2023-12-17 20:25:36.585139 DEBUG keystoneauth.session RESP BODY: {"versions": {"values": [{"id": "v3.14", "status": "stable", "updated": "2020-04-07T00:00:00Z", "links": [{"rel": "self", "href": "http://192.168.1.101/identity/v3/"}], "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}]}]}}
2023-12-17 20:25:36.585193 DEBUG keystoneauth.session GET call to http://192.168.1.101/identity used request id req-bc1f979a-d01a-4e7a-b486-bc446860215d
2023-12-17 20:25:36.585491 DEBUG keystoneauth.identity.v3.base Making authentication request to http://192.168.1.101/identity/v3/auth/tokens
2023-12-17 20:25:36.600377 DEBUG keystoneauth.session Request returned failure status: 404
devstack@keystone:
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f979a-d01a-4e7a-b486-bc446860215d None None] REQUEST_METHOD: `GET` {{(pid=2308208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f979a-d01a-4e7a-b486-bc446860215d None None] SCRIPT_NAME: `/identity` {{(pid=2308208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308208]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1f979a-d01a-4e7a-b486-bc446860215d None None] PATH_INFO: `/` {{(pid=2308208) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308208]: [pid: 2308208|app: 0|req: 1036/2062] 192.168.1.101 () {60 vars in 1015 bytes} [Sun Dec 17 15:25:36 2023] GET /identity => generated 272 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 236 bytes (1 switches on core 0)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6003b1d8-4379-491c-b146-7736efb37b04 None None] Authenticating user token {{(pid=2308207) process_request /usr/local/lib/python3.9/site-packages/keystonemiddleware/auth_token/__init__.py:406}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.common.fernet_utils [None req-6003b1d8-4379-491c-b146-7736efb37b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=2308207) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: INFO keystone.server.flask.request_processing.middleware.auth_context [None req-6003b1d8-4379-491c-b146-7736efb37b04 None None] Invalid user token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640ad9a-a9e5-4aff-b9dd-cda3908a873f None None] REQUEST_METHOD: `POST` {{(pid=2308207) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640ad9a-a9e5-4aff-b9dd-cda3908a873f None None] SCRIPT_NAME: `/identity` {{(pid=2308207) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1640ad9a-a9e5-4aff-b9dd-cda3908a873f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=2308207) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: DEBUG keystone.common.fernet_utils [None req-1640ad9a-a9e5-4aff-b9dd-cda3908a873f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=2308207) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}}
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application [None req-1640ad9a-a9e5-4aff-b9dd-cda3908a873f None None] Could not recognize Fernet token: keystone.exception.TokenNotFound: Could not recognize Fernet token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application Traceback (most recent call last):
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/token_formatters.py", line 89, in unpack
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self.crypto.decrypt(token.encode('utf-8'))
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib64/python3.9/site-packages/cryptography/fernet.py", line 211, in decrypt
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application raise InvalidToken
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application cryptography.fernet.InvalidToken
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application Traceback (most recent call last):
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/providers/fernet/core.py", line 99, in validate_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self.token_formatter.validate_token(token_id)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/token_formatters.py", line 173, in validate_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application serialized_payload = self.unpack(token)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/token_formatters.py", line 91, in unpack
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application raise exception.ValidationError(
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application keystone.exception.ValidationError: Could not recognize Fernet token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application Traceback (most recent call last):
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1823, in full_dispatch_request
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application rv = self.dispatch_request()
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 489, in wrapper
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/flask/views.py", line 107, in view
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/server/flask/common.py", line 1064, in wrapper
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return f(*args, **kwargs)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 316, in post
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application token = authentication.authenticate_for_token(auth_data)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 185, in authenticate_for_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application authenticate(auth_info, auth_context)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 152, in authenticate
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application resp = method.authenticate(auth_info.get_method_data(method_name))
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/auth/plugins/token.py", line 42, in authenticate
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application token = self._get_token_ref(auth_payload)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/auth/plugins/token.py", line 36, in _get_token_ref
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return PROVIDERS.token_provider_api.validate_token(token_id)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 110, in wrapped
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 146, in validate_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application token = self._validate_token(token_id)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/decorator.py", line 232, in fun
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/cache/region.py", line 1594, in get_or_create_for_user_func
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self.get_or_create(
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/cache/region.py", line 1054, in get_or_create
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application with Lock(
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/lock.py", line 185, in __enter__
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self._enter()
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/lock.py", line 94, in _enter
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/lock.py", line 178, in _enter_create
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application return self.creator()
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.9/site-packages/dogpile/cache/region.py", line 1007, in gen_value
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application created_value = creator(
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 159, in _validate_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application issued_at, expires_at) = self.driver.validate_token(token_id)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/providers/fernet/core.py", line 101, in validate_token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application raise exception.TokenNotFound(e)
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Could not recognize Fernet token
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: ERROR keystone.server.flask.application
Dec 17 15:25:36 gthiemon-devstack <email address hidden>[2308207]: [pid: 2308207|app: 0|req: 1027/2063] 192.168.1.101 () {66 vars in 1137 bytes} [Sun Dec 17 15:25:36 2023] POST /identity/v3/auth/tokens => generated 88 bytes in 12 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0)
The bug is due to "auth tokens are deleted when using OpenStackConfig" https://bugs.launchpad.net/openstacksdk/+bug/2046645
A workaround is possible in octavia-dashboard, passing an 'auth_token' attribute instead of 'token' attribute at [0] would fix the issue
[0] https://opendev.org/openstack/octavia-dashboard/src/branch/master/octavia_dashboard/sdk_connection.py#L46
Fix proposed to branch: master
Review: https:/