OpenStack Identity (keystone)

*-grenade-multinode jobs fail with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte

Bug #1891244 reported by melanie witt
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Undecided
melanie witt

Bug Description

With a recent bump of the upper constraint for the msgpack library to version v1.0 [1], the *-grenade-multinode gate jobs fail with the following trace.

The data format was changed in msgpack v1.0 [2], so during a rolling upgrade, tokens cached from the N-1 release (msgpack < v1.0) will fail to unpack with the new format in msgpack v1.0.

Aug 11 14:45:47.321414 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239fb20e-9ca7-476c-ba19-4c6707192097 None None] Validating token access rules against request {{(pid=29023) validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545}}
Aug 11 14:45:47.325427 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] RBAC: auth_context: {'token': <TokenModel (audit_id=XBmEG0BZRNCTDw5-ANYeHw, audit_chain_id=['XBmEG0BZRNCTDw5-ANYeHw']) at 0x7f6d17d4d0f0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60203c873e874a11ae4e8ee43f6ed7cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cdb7b38e6fc84c20b1aa05d012d599f7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=29023) fill_context /opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}}
Aug 11 14:45:47.325958 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] REQUEST_METHOD: `GET` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
Aug 11 14:45:47.326214 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] SCRIPT_NAME: `/identity` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
Aug 11 14:45:47.326462 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
Aug 11 14:45:47.327842 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: DEBUG keystone.common.fernet_utils [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=29023) load_keys /opt/stack/new/keystone/keystone/common/fernet_utils.py:289}}
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: CRITICAL keystone [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Unhandled error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone Traceback (most recent call last):
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.wsgi_app(environ, start_response)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.app(environ, start_response)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = req.get_response(self.application)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone application, catch_exc_info=False)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 143, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return resp(environ, start_response)
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = req.get_response(self.application)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone application, catch_exc_info=False)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/osprofiler/web.py", line 112, in __call__
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return request.get_response(self.application)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone application, catch_exc_info=False)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = req.get_response(self.application)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone application, catch_exc_info=False)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.app(environ, start_response)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = req.get_response(self._app)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone application, catch_exc_info=False)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return app(environ, start_response)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = self.handle_exception(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone [Previous line repeated 27 more times]
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone reraise(exc_type, exc_value, tb)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone raise value.with_traceback(tb)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone response = self.full_dispatch_request()
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone rv = self.handle_user_exception(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return original_handler(e)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone [Previous line repeated 27 more times]
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone reraise(exc_type, exc_value, tb)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone raise value.with_traceback(tb)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone rv = self.dispatch_request()
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.view_functions[rule.endpoint](**req.view_args)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 468, in wrapper
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = resource(*args, **kwargs)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.dispatch_request(*args, **kwargs)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone resp = meth(*args, **kwargs)
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/api/auth.py", line 285, in get
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone ENFORCER.enforce_call(action='identity:validate_token')
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 414, in enforce_call
Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone subj_token_target_data = cls._extract_subject_token_target_data()
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 256, in _extract_subject_token_target_data
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone access_rules_support=access_rules_support
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/manager.py", line 115, in wrapped
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone __ret_val = __f(*args, **kwargs)
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 145, in validate_token
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone token = self._validate_token(token_id)
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "<decorator-gen-26>", line 2, in _validate_token
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 1357, in get_or_create_for_user_func
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone key, user_func, timeout, should_cache_fn, (arg, kw)
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 959, in get_or_create
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone async_creator,
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 185, in __enter__
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self._enter()
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 94, in _enter
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone generated = self._enter_create(value, createdtime)
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 178, in _enter_create
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.creator()
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 913, in gen_value
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone *creator_args[0], **creator_args[1]
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 158, in _validate_token
Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone expires_at) = self.driver.validate_token(token_id)
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/providers/fernet/core.py", line 96, in validate_token
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone return self.token_formatter.validate_token(token_id)
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/token_formatters.py", line 173, in validate_token
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone versioned_payload = msgpack.unpackb(serialized_payload)
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 <email address hidden>[29022]: ERROR keystone

[1] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373
[2] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/745752

Changed in keystone:
assignee: nobody → melanie witt (melwitt)
status: New → In Progress
Revision history for this message
Matthew Thode (prometheanfire) wrote :

subscribed, but reqs can't be added as a project because it's not on LP anymore :|

I do hope that this will be actively worked on though (since it's blocking adoption by the rest of openstack as well.

Revision history for this message
melanie witt (melwitt) wrote :

I have proposed a fix for keystone, linked in comment #1.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.opendev.org/745752
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7d6c71ba26694c21110280e741b9ffe2d36a94ca
Submitter: Zuul
Branch: master

commit 7d6c71ba26694c21110280e741b9ffe2d36a94ca
Author: melanie witt <email address hidden>
Date: Tue Aug 11 21:19:01 2020 +0000

    Support format for msgpack < 1.0 in token formatter

    msgpack v1.0 changed its data format [1] and during a rolling upgrade, attempts
    to unpack cached tokens with old data format with the new default raw=False
    result in the following error:

      UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in
      position 3: invalid start byte

    This passes raw=True to support backward-compat with the old format
    until we are guaranteed to have msgpack >= 1.0 in the N-1 release of
    a rolling upgrade.

    Closes-Bug: #1891244

    [1]
    https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10

    Change-Id: I6c61df6c097fef698c659c79402c4381ec7f3586

Changed in keystone:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.