OVB jobs on vexxhost are failing to create the stack - CREATE failed: WaitConditionFailure: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: Signal 1 received

[Ronelle Landy]

OVB jobs running on vexxhost fail when creating the OVB stack:

2021-09-30 15:11:36.595915 | primary | | stack_status | CREATE_FAILED |
2021-09-30 15:11:36.595936 | primary | | stack_status_reason | Resource CREATE failed: WaitConditionFailure: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: Signal 1 received |
2021-09-30 15:11:36.595953 | primary | | parameters | OS::project_id: 4b633c451ac74233be3721a3635275e5 |

Full example log is below:

https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/43b517f/job-output.txt

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001

failure started around 12pm UTC

[Ronelle Landy (rlandy)]

BMC log:

https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/43b517f/logs/bmc-console.log

-----END SSH HOST KEY KEYS-----
[ 52.497063] cloud-init[1173]: Cloud-init v. 18.5 finished at Thu, 30 Sep 2021 15:10:56 +0000. Datasource DataSourceConfigDrive [net,ver=2][source=/dev/sr0]. Up 52.49 seconds
[[1;31mFAILED[0m] Failed to start Execute cloud user/final scripts.
See 'systemctl status cloud-final.service' for details.
[[32m OK [0m] Reached target Cloud-init target.

[Ronelle Landy (rlandy)]

[ 40.682184] cloud-init[1156]: Fetching private network
[ 40.683882] cloud-init[1156]: Traceback (most recent call last):
[ 40.685735] cloud-init[1156]: File "<stdin>", line 13, in <module>
[ 40.687497] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 1350, in list
[ 40.692083] cloud-init[1156]: microversion=microversion)
[ 40.693708] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 375, in get
[ 40.698299] cloud-init[1156]: return self.request(url, 'GET', **kwargs)
[ 40.700167] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/openstack/_adapter.py", line 124, in request
From BMC log:

[ 40.714155] cloud-init[1156]: **kwargs)
[ 40.715588] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 237, in request
[ 40.718137] cloud-init[1156]: return self.session.request(url, method, **kwargs)
[ 40.720049] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 835, in request
[ 40.756347] cloud-init[1156]: resp = send(**kwargs)
[ 40.758394] cloud-init[1156]: File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 930, in _send_request
[ 40.761164] cloud-init[1156]: raise exceptions.SSLError(msg)
[ 40.762770] cloud-init[1156]: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://network.rdo.mtl2.vexxhost.net/v2.0/networks: HTTPSConnectionPool(host='network.rdo.mtl2.vexxhost.net', port=443): Max retries exceeded with url: /v2.0/networks?name=private-811973_2_97221 (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),))

[Ronelle Landy (rlandy)]

NOTE: vexxhost change:

<dpawlik> mnaser: hey, today I create a ticket for update the DNS to our subnets for public network
<dpawlik> #NHU-353717

[Ronelle Landy (rlandy)]

Ticket to vexxhost:

ticket is #QMX-351850

[Ronelle Landy (rlandy)]

Uploaded new bmc-template image