Multiple NODE_FAILURE on all jobs running on rdoproject.org tenant

[chandan kumar]

Almost all the centos stream 9 and 8 jobs running on vexxhost cloud are hitting node_failure.
It impacted RDO third party check jobs, master and wallaby integration/component lines.
Most of them are cs9 based jobs.

Based on https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE&skip=0 result page.

Based on https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-9-wallaby-promote-promoted-components-to-tripleo-ci-testing and https://review.rdoproject.org/zuul/build/6943583d6a5548098448393a33702224

periodic-tripleo-centos-9-wallaby-promote-promoted-components-to-tripleo-ci-testing NODE_FAILURE started at 2022-08-13 21:11:15 UTC.

and last one while filling this bug:
tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001 - 2022-08-15 06:23:57 UTC
- https://review.rdoproject.org/zuul/build/271a4c0e1eef415494bc979c372c0258

It might be happening from very long time.
It is a tracker bug for the same.

Thank you Marios for adding more details in https://bugs.launchpad.net/tripleo/+bug/1986502/comments/2 .

[chandan kumar (chkumar246)]

Based on nodepool logs: https://softwarefactory-project.io/nodepool-builder/nodepool-builder.softwarefactory-project.io/?C=M;O=D

Last cs9 image build was on https://softwarefactory-project.io/nodepool-builder/nodepool-builder.softwarefactory-project.io/cloud-centos-9-stream-tripleo-no-update-0000003685.log - 2022-08-14 11:23

But there is no error shown in the logs.

[Marios Andreou (marios-b)]

the latest buildsets from the main lines master/wallaby/train are all from before the weekend:

master [1] wallaby/9 [2] train [3]

So this bug is blocking all lines. The promote to tripleo-ci-testing jobs are hitting this bug e.g. see [4] and attached screenshot here so none of the jobs can run without a new tripleo-ci-testing

[1] https://review.rdoproject.org/zuul/buildset/953eacd0246648ada324268a4c21a8e6
[2] https://review.rdoproject.org/zuul/buildset/41e058e884fc4e35bfe4bd19d8471411
[3] https://review.rdoproject.org/zuul/buildset/0a685aa16ac04f0eb43c04c925bfd36b
[4] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-9-wallaby-promote-promoted-components-to-tripleo-ci-testing

[Sandeep Yadav (sandeepyadav93)]

Issue resolved after changing OS_AUTHURL.