Scenario001 standalone train is failing in tempestconf setup - Request on service 'image' with url 'http://192.168.24.1:9292' failed. Connection refused.

[Douglas Viroel]

Scenario001 standalone train is failing since 08/11:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-train

Tempest conf log shows Connection Refused error:

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz

Errors file:
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/var/log/extra/errors.txt.gz

tempestconf Error:
ERROR config_tempest.constants [-] Request on service 'image' with url 'http://192.168.24.1:9292' failed: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=9292): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fccdbe77940>: Failed to establish a new connection: [Errno 111] Connection refused',))[00m
CRITICAL tempest [-] Unhandled error: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.24.1', port=9292): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fccdbe77940>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
   File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
     (self._dns_host, self.port), self.timeout, **extra_kw
   File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
     raise err
   File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
     sock.connect(sa)
 ConnectionRefusedError: [Errno 111] Connection refused

[Cédric Jeanneret (cjeanner)]

Seems to be linked to a crashed Glance container?
What's supposed to be listening on :9292 ?
According to this log file (from the same job), it sounds like glance is crashed:
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/var/log/extra/failed_containers.log.txt.gz

Also, there are issues related to cinder and permissions here:
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/var/log/extra/errors.txt.gz

[Cédric Jeanneret (cjeanner)]

namely:
7c60ab493bfa 192.168.24.1:8787/tripleotraincentos8/centos-binary-glance-api:3dbd1c4a2703757e335151b227162b9a-updated-20210811191725 kolla_start 6 minutes ago Exited (99) Less than a second ago glance_api

2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service [-] Error starting thread.: tooz.coordination.ToozConnectionError: [Errno 13] Permission denied: '/var/lib/cinder/groups'
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service Traceback (most recent call last):
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/tooz/drivers/file.py", line 278, in _start
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service fileutils.ensure_tree(a_dir)
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/oslo_utils/fileutils.py", line 42, in ensure_tree
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service os.makedirs(path, mode)
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib64/python3.6/os.py", line 220, in makedirs
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service mkdir(name, mode)
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service PermissionError: [Errno 13] Permission denied: '/var/lib/cinder/groups'
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service Traceback (most recent call last):
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 810, in run_service
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service service.start()
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/cinder/service.py", line 220, in start
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service coordination.COORDINATOR.start()
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log: 47 ERROR oslo_service.service File "/usr/lib/python3.6/site-packages/cinder/coordination.py", line 67, in start
2021-08-11 20:04:41.253 ERROR /var/log/containers/cinder/cinder-backup.log:...

[Cédric Jeanneret (cjeanner)]

While digging a bit deeper in the logs, we can see glance is having a hard time:

At the end of the following log:
https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/1c1f56b/logs/undercloud/var/log/containers/glance/api.log.txt.gz

we can see this ERROR trace:

021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd [-] Error connecting to ceph cluster.: rados.PermissionDeniedError: [errno 13] error connecting to the cluster
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd Traceback (most recent call last):
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd File "/usr/lib/python3.6/site-packages/glance_store/_drivers/rbd.py", line 273, in get_connection
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd client.connect(timeout=self.connect_timeout)
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd File "rados.pyx", line 925, in rados.Rados.connect
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd rados.PermissionDeniedError: [errno 13] error connecting to the cluster
2021-08-16 20:33:12.278 8 ERROR glance_store._drivers.rbd
2021-08-16 20:33:12.280 8 DEBUG glance_store.capabilities [-] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3.6/site-packages/glance_store/capabilities.py:95

[yatin (yatinkarel)]

https://github.com/openstack/glance_store/commit/a6227665ef8ec654ca342362996b1cc40357a290 is needed to clear the issue which is not available in last tag release of glance-store in Train.
This issue triggered after glance-store was pinned https://review.rdoproject.org/r/c/rdoinfo/+/34888 for another issue https://bugs.launchpad.net/tripleo/+bug/1938558. Since glance store is following upper-constraints now, so we would need new tag release for it and get it available in Train upper-constraints to clear this issue, or if that takes time can pin to required commit in rdoinfo.

[yatin (yatinkarel)]

<<so we would need new tag release for it
glance store is already tagged EM https://review.opendev.org/c/openstack/releases/+/790745, so can't get it in upstream releases, so need to handle it in RDO itself https://review.rdoproject.org/r/c/rdoinfo/+/34975.

[Giulio Fidente (gfidente)]

I understand the fix at [1] will make Glance work regardless of if any of the stores is misconfigured but I am not sure which one of the stores in this CI job would be misconfigured and why? Cyril can you help debug the logs to see if we can fix it? We have glance_api at [2] and ceph conf at [3]

1. https://github.com/openstack/glance_store/commit/a6227665ef8ec654ca342362996b1cc40357a290
2. https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf.txt.gz
3. https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/64468c8/logs/undercloud/etc/ceph/

[Giulio Fidente (gfidente)]

the rbd store is most probably failing because the "images" pool wasn't created in the ceph cluster; we're working on that via https://review.opendev.org/c/openstack/tripleo-heat-templates/+/805029

I am not sure if we want to "related" that fix to this bug?