fs001 and fs035 OVB have been largely unstable for about a month - across multiple releases.
There are various errors being watch (including node provision and overcloud deploy failures), however many test are failing at the tempest stage - with, what looks like, related connection errors.
On CentOS 8 releases(wallaby and train), we see the following errors:
ft1.1: setUpClass (keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest)testtools.testresult.real._StringException: Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/test.py", line 182, in setUpClass
....
'Unexpected status code {0}'.format(resp.status))
tempest.lib.exceptions.IdentityError: Got identity error
Details: Unexpected status code 500
Example logs:
https://logserver.rdoproject.org/55/36255/67/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/9967e52/logs/undercloud/var/log/tempest/stestr_results.html.gz
https://logserver.rdoproject.org/62/42462/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/9da7633/logs/undercloud/var/log/tempest/stestr_results.html.gz
https://logserver.rdoproject.org/91/42491/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/626ac5d/logs/undercloud/var/log/tempest/stestr_results.html.gz
with related logs:
https://logserver.rdoproject.org/55/36255/67/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/9967e52/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz
https://logserver.rdoproject.org/55/36255/67/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/9967e52/logs/overcloud-controller-0/var/log/containers/haproxy/haproxy.log.txt.gz
On CentOS 9 releases:
We see different - but possibly related - errors:
https://logserver.rdoproject.org/55/36255/67/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/498eb76/logs/undercloud/var/log/tempest/stestr_results.html.gz
File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
These errors are preventing promotions of wallaby and train c8 and delaying the master c9 promotion.
For additional info: general OVB stats:
job_name, success, failure tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-wallaby 0 10 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001-wallaby 1 13 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp_1supp- featureset064- wallaby 16 22 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp_1supp- featureset039- wallaby 15 27 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t035-wallaby 10 29 tripleo- ci-centos- 9-ovb-1ctlr_ 2comp-featurese t020-wallaby 15 31 tripleo- ci-centos- 9-ovb-1ctlr_ 1comp-featurese t002-wallaby 40 7 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-wallaby 6 40 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp_1supp- featureset064- master 6 31 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp_1supp- featureset039- master 4 33 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t035-master 3 42 tripleo- ci-centos- 9-ovb-1ctlr_ 2comp-featurese t020-master 5 43 tripleo- ci-centos- 9-ovb-3ctlr_ 1comp-featurese t001-master 7 39 tripleo- ci-centos- 9-ovb-1ctlr_ 1comp-featurese t002-master 19 32 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp_1supp- featureset039- train 18 10 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-train 1 19 tripleo- ci-centos- 8-ovb-1ctlr_ 2comp-featurese t020-train 14 16 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001-train 5 25 tripleo- ci-centos- 8-ovb-1ctlr_ 1comp-featurese t002-train 21 7 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp_1supp- featureset039- victoria 6 7 tripleo- ci-centos- 8-ovb-1ctlr_ 1comp-featurese t002-victoria 10 6 tripleo- ci-centos- 8-ovb-1ctlr_ 2comp-featurese t020-victoria 0 17 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t035-victoria 0 13 tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001-victoria 2 14
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-
periodic-