Description:-
Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy.
Jobs are failing since 08th Nov'20
Logs:-
https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/b8ce7e2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
~~~
2020-11-09 08:34:36.962241 | fa163e7d-71c4-2223-cd8b-0000000046fc | TASK | Check containers status
[ERROR]: Container(s) which finished with wrong return code:
['nova_wait_for_compute_service']
[ERROR]: Container(s) which failed to be created by podman_container module:
['nova_wait_for_compute_service']
2020-11-09 08:34:38.744478 | fa163e7d-71c4-2223-cd8b-0000000046fc | FATAL | Check containers status | standalone | error={"changed": false, "msg": "Failed container(s): ['nova_wait_for_compute_service'], check logs in /var/log/containers/stdouts/"}
2020-11-09 08:34:38.746197 | fa163e7d-71c4-2223-cd8b-0000000046fc | TIMING | tripleo_container_manage : Check containers status | standalone | 0:41:25.278908 | 1.78s
PLAY RECAP *********************************************************************
standalone : ok=640 changed=292 unreachable=0 failed=1 skipped=209 rescued=0 ignored=0
~~~
* INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register - as final log - "Nova-compute service registered" not noticed in logs
https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/b8ce7e2/logs/undercloud/var/log/extra/podman/containers/nova_wait_for_compute_service/stdout.log.txt.gz
~~~
DEBUG:keystoneauth.session:GET call to compute for http://192.168.24.3:8774/v2.1/os-services?binary=nova-compute used request id req-13c1c788-99ce-49c1-a0ae-25dc23a410dc
INFO:nova_wait_for_compute_service:Waiting for nova-compute service to register
~~~
* https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/b8ce7e2/logs/undercloud/var/log/extra/errors.txt.txt.gz
~~~
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova Traceback (most recent call last):
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 350, in get_or_set_cached_cell_and_set_connections
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova cell_tuple = CELL_CACHE[cell_mapping.uuid]
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova KeyError: '00000000-0000-0000-0000-000000000000'
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova During handling of the above exception, another exception occurred:
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova Traceback (most recent call last):
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova sys.exit(main())
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/compute.py", line 59, in main
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova topic=compute_rpcapi.RPC_TOPIC)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 252, in create
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova utils.raise_if_old_compute()
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/utils.py", line 1068, in raise_if_old_compute
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova ctxt, ['nova-compute'])
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/objects/service.py", line 554, in get_minimum_version_all_cells
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova binaries)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 545, in scatter_gather_all_cells
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova fn, *args, **kwargs)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 433, in scatter_gather_cells
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova with target_cell(context, cell_mapping) as cctxt:
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova return next(self.gen)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 393, in target_cell
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova set_target_cell(cctxt, cell_mapping)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 366, in set_target_cell
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova get_or_set_cached_cell_and_set_connections()
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 360, in inner
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova return f(*args, **kwargs)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/context.py", line 354, in get_or_set_cached_cell_and_set_connections
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova db_connection_string)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 79, in create_context_manager
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova return IMPL.create_context_manager(connection=connection)
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/common.py", line 48, in __call__
2020-11-09 08:24:02.769 ERROR /var/log/containers/nova/nova-compute.log: 8 ERROR nova raise exception.DBNotAllowed(binary=service_name)
~~~
Another example:-
https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario012-standalone-compute-master/cde6346/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
Observation:-
Looks like it started after the merge of https://review.opendev.org/#/c/738482
puppet promotion jobs for master are also failing post https:/ /review. opendev. org/#/c/ 738482. /logserver. rdoproject. org/ci. centos. org/weirdo- generic- puppet- openstack- scenario001/ 17324/weirdo- project/ puppet. txt.gz /logserver. rdoproject. org/ci. centos. org/weirdo- generic- puppet- openstack- scenario002/ 17266/weirdo- project/ puppet. txt.gz
https:/
https:/