tempest tests are all failing on all OVB jobs in c8/c9 master

[Ronelle Landy]

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master and periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master and all fs001 jobs in the component lines are failing all tempest tests.

Logs are included below:

https://logserver.rdoproject.org/56/36256/44/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master/be79af9/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/44cdfac/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/53/21853/42/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/43e33cb/logs/undercloud/var/log/tempest/stestr_results.html.gz

Both the c8 and c9 tests were impacted by mirror issues or other bugs and did not run to tempest for a few days.

The error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable
....
  conn = self._new_conn()
  File "/usr/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5f23557340>: Failed to establish a new connection: [Errno 101] Network is unreachable

[Ronelle Landy (rlandy)]

Last passing tests, we see on 01/31:

 https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master&job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/56/36256/44/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-master/be79af9/logs/undercloud/var/log/containers/stdouts/neutron_api.log.txt.gz

2022-02-03T17:40:15.811635221+00:00 stderr P keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to http://192.168.24.3:5000: HTTPConnectionPool(host='192.168.24.3', port=5000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe96e71e2b0>: Failed to establish a new connection:
2022-02-03T17:40:15.811974520+00:00 stderr F [Errno 111] ECONNREFUSED',))
2022-02-03T17:40:15.811974520+00:00 stderr F Removing descriptor: 15

[Ronelle Landy (rlandy)]

Notes from IRC:

<mwhahaha> undercloud won't have access to internal_api
<mwhahaha> which is why it was always ctlplane

<mwhahaha> https://github.com/openstack/tempest/blob/master/tempest/config.py#L155
<mwhahaha> need to configure v3_endpoint_type = external
<mwhahaha> it defaults to admin
<mwhahaha> since we switched it, that's no longer ctlplane

<mwhahaha> so yea https://opendev.org/openstack/tripleo-heat-templates/commit/ad97bd70592a57708313d2b87a61f0be01c7b004 broke it
<mwhahaha> but the fix is to set the correct endpoint type in tempest
<mwhahaha> because we should be using external for tempest from UC

[Ronelle Landy (rlandy)]

so in our case, that lands up being public

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827767

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Harald Jensås <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827767
Reason: arxcruz> hjensas yup, validate-tempest is deprecated

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/827779
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/67cf0fbabc101affc3b53dee3e23e4f66946073d
Submitter: "Zuul (22348)"
Branch: master

commit 67cf0fbabc101affc3b53dee3e23e4f66946073d
Author: Arx Cruz <email address hidden>
Date: Fri Feb 4 00:57:27 2022 +0100

    Add identity.v3_endpoint_type to extra facts in tempestconf

    Some tests are failing because it's using the adminURL instead of
    publicURL. This patch add the config option to tempestconf to force the
    use of the publicURL.

    Change-Id: I3d5146041df88fbcaad6b0f6d6dcb05d734251b0
    Related-Bug: #1959930