tripleo

standalone-full-tempest-api-network master/zed are failing - Could not find service: keystone.

Bug #2000222 reported by Amol Kahat on 2022-12-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Invalid	Low	Unassigned	tripleo antelope-1

Bug Description

Description
===========

Jobs are failing on tempest test. Identity error

```
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application [None req-f2e29eb8-9921-4af5-94e9-4e9e460ad548 1b03a4b0408d4e69947e9fc2c7c1a456 - - all default -] Could not find service: keystone.: keystone.exception.ServiceNotFound: Could not find service: keystone.
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application Traceback (most recent call last):
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/catalog/core.py", line 152, in get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.driver.get_service(service_id)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/catalog/backends/sql.py", line 196, in get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict()
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/catalog/backends/sql.py", line 191, in _get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: keystone.
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application During handling of the above exception, another exception occurred:
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application Traceback (most recent call last):
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application rv = self.dispatch_request()
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 467, in wrapper
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application resp = resource(*args, **kwargs)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/flask/views.py", line 83, in view
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.dispatch_request(*args, **kwargs)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application resp = meth(*args, **kwargs)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/api/services.py", line 45, in get
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self._get_service(service_id)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/api/services.py", line 34, in _get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id))
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/common/manager.py", line 115, in wrapped
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "<decorator-gen-17>", line 2, in get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.get_or_create(
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application with Lock(
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/lock.py", line 185, in __enter__
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self._enter()
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/lock.py", line 94, in _enter
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/lock.py", line 178, in _enter_create
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application return self.creator()
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/dogpile/cache/region.py", line 995, in gen_value
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application created_value = creator(
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application File "/usr/lib/python3.9/site-packages/keystone/catalog/core.py", line 154, in get_service
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id)
2022-12-20 12:48:02.523 23 ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: keystone.
```

Logs
====
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/d7ba2d2/logs/undercloud/var/log/containers/keystone/keystone.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/47a4ec1/logs/undercloud/var/log/containers/keystone/keystone.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-zed/581ab83/logs/undercloud/var/log/containers/keystone/keystone.log.txt.gz

See original description

Tags:

Amol Kahat (amolkahat) on 2022-12-21

description:

updated

Revision history for this message

Ronelle Landy (rlandy) wrote on 2022-12-21 (last edit on 2022-12-21):

Checking latest:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master&skip=0

three failing master runs - and then a pass in testproject

wallaby- looking green: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby&skip=0

zed is starting to look more green:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-zed&skip=0

summary:	- Could not find service: keystone. + standalone-full-tempest-api-network master/zed are failing - Could not + find service: keystone.
summary:	- standalone-full-tempest-api-network master/zed are failing - Could not - find service: keystone. + standalone-full-tempest-api-network master/zed/wallaby are failing - + Could not find service: keystone.
Changed in tripleo:
status:	Confirmed → Triaged
summary:	- standalone-full-tempest-api-network master/zed/wallaby are failing - - Could not find service: keystone. + standalone-full-tempest-api-network master/zed are failing - Could not + find service: keystone.