queens and rocky CI - Source /home/zuul/.config/openstack/clouds.yaml not found

[Harald Jensås]

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ba5/771543/1/check/tripleo-ci-centos-7-undercloud-oooq/ba5d6f2/job-output.txt

2021-01-20 09:50:45.037344 | primary | TASK [Copy clouds.yaml file from user's home to /root directory] ***************
2021-01-20 09:50:45.037635 | primary | Wednesday 20 January 2021 09:50:45 +0000 (0:00:00.884) 0:01:05.466 *****
2021-01-20 09:50:47.396914 | primary | fatal: [undercloud]: FAILED! => {
2021-01-20 09:50:47.397700 | primary | "changed": false
2021-01-20 09:50:47.397799 | primary | }
2021-01-20 09:50:47.397843 | primary |
2021-01-20 09:50:47.397880 | primary | MSG:
2021-01-20 09:50:47.397943 | primary |
2021-01-20 09:50:47.398080 | primary | Source /home/zuul/.config/openstack/clouds.yaml not found
2021-01-20 09:50:47.412931 | primary |
2021-01-20 09:50:47.413638 | primary | TASK [Copy clouds.yaml file /etc/openstack to /root directory] *****************
2021-01-20 09:50:47.413730 | primary | Wednesday 20 January 2021 09:50:47 +0000 (0:00:02.373) 0:01:07.839 *****
2021-01-20 09:50:49.587783 | primary | fatal: [undercloud]: FAILED! => {
2021-01-20 09:50:49.588109 | primary | "changed": false
2021-01-20 09:50:49.588146 | primary | }
2021-01-20 09:50:49.588175 | primary |
2021-01-20 09:50:49.588209 | primary | MSG:
2021-01-20 09:50:49.588237 | primary |
2021-01-20 09:50:49.588301 | primary | Source /etc/openstack/clouds.yaml not found

[Harald Jensås (harald-jensas)]

Removed duplicate of https://bugs.launchpad.net/tripleo/+bug/1911696 as it's fix has been released, but the problem remains.

2021-01-21 10:32:50.976664 | primary | TASK [Copy clouds.yaml file from user's home to /root directory] ***************
2021-01-21 10:32:50.977002 | primary | Thursday 21 January 2021 10:32:50 +0000 (0:00:00.756) 0:01:00.475 ******
2021-01-21 10:32:52.975408 | primary | fatal: [undercloud]: FAILED! => {
2021-01-21 10:32:52.975673 | primary | "changed": false
2021-01-21 10:32:52.975739 | primary | }
2021-01-21 10:32:52.975805 | primary |
2021-01-21 10:32:52.975873 | primary | MSG:
2021-01-21 10:32:52.975930 | primary |
2021-01-21 10:32:52.976091 | primary | Source /home/zuul/.config/openstack/clouds.yaml not found
2021-01-21 10:32:52.999150 | primary |
2021-01-21 10:32:52.999325 | primary | TASK [Copy clouds.yaml file /etc/openstack to /root directory] *****************
2021-01-21 10:32:52.999448 | primary | Thursday 21 January 2021 10:32:52 +0000 (0:00:02.022) 0:01:02.497 ******
2021-01-21 10:32:55.049652 | primary | fatal: [undercloud]: FAILED! => {
2021-01-21 10:32:55.050234 | primary | "changed": false
2021-01-21 10:32:55.050275 | primary | }

[Harald Jensås (harald-jensas)]

This also happens in Rocky: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1b5/771542/1/check/tripleo-ci-centos-7-undercloud-containers/1b5c785/job-output.txt

2021-01-20 09:09:06.429058 | primary | Source /home/zuul/.config/openstack/clouds.yaml not found
2021-01-20 09:09:06.442375 | primary |
2021-01-20 09:09:06.442601 | primary | TASK [Copy clouds.yaml file /etc/openstack to /root directory] *****************
2021-01-20 09:09:06.442764 | primary | Wednesday 20 January 2021 09:09:06 +0000 (0:00:02.734) 0:00:15.456 *****
2021-01-20 09:09:08.587955 | primary | fatal: [undercloud]: FAILED! => {
2021-01-20 09:09:08.588368 | primary | "changed": false
2021-01-20 09:09:08.588438 | primary | }
2021-01-20 09:09:08.588490 | primary |
2021-01-20 09:09:08.588575 | primary | MSG:
2021-01-20 09:09:08.588625 | primary |
2021-01-20 09:09:08.588766 | primary | Source /etc/openstack/clouds.yaml not found

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Can you check if that patch was in https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 in those failures ?

[Harald Jensås (harald-jensas)]

Given that https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837 merged Jan 15,
and I saw these errors Jan 20 and again Jan 21 I assume the patch merged 5 days earlier was used? (If not, we have a problem with CI ...)

Job logs says commit 53b4418236eaa82396cfeaab747a960c3b48d58a was the last commit to tripleo-quickstart-extras:

2021-01-21 09:30:06.764292 | primary | opendev.org/openstack/tripleo-quickstart-extras checked out to:
2021-01-21 09:30:06.764415 | primary | 53b4418236eaa82396cfeaab747a960c3b48d58a Merge "Increase timeout for containers build"

$ git log 53b4418236eaa82396cfeaab747a960c3b48d58a

commit 53b4418236eaa82396cfeaab747a960c3b48d58a (HEAD -> master, origin/master, origin/HEAD, gerrit/master)
Merge: 84780c71 b777aba5
Author: Zuul <email address hidden>
Date: Wed Jan 20 11:59:56 2021 +0000

    Merge "Increase timeout for containers build"

commit b777aba53a5ef5b52afd257231e0cd6a472318d3
Author: Sagi Shnaidman <email address hidden>
Date: Thu Jan 7 17:53:39 2021 +0200

    Increase timeout for containers build

    In some environments[1] it takes time to build containers, let's
    increase the timeout. Disable task retries because we have retries
    with tenacity and we can't keep previous logs.

    Change-Id: I793e44fc561b08776c7f8e145887cfeae08d8e5b

commit 84780c715cf71295d15c1addfeb91d6a71e2678d
Author: Ronelle Landy <email address hidden>
Date: Mon Jan 11 14:54:23 2021 -0500

    Correct forwarder_address variable definition check

    forwarder_addresses_output is the correct variable
    name to check when verifying if unbound forwarders
    are defined.

    Change-Id: I9dddafebee6732ff73b1024d58fb789869571f50

commit 1651d8de797f594045906c3d9b21e48ed5445ff8
Merge: a9ec928c b2a36325
Author: Zuul <email address hidden>
Date: Fri Jan 15 11:43:46 2021 +0000

    Merge "Set condition to run os_tempest"

   ^^ Patch is there.

[wes hayutin (weshayutin)]

See it here:
https://18eaa349c9388c8b78c5-fa065b3d3badff7cb87ccafc6f749b5a.ssl.cf5.rackcdn.com/770266/5/check/tripleo-ci-centos-7-undercloud-containers/81223f4/job-output.txt

[Arx Cruz (arxcruz)]

This should be fixed by https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/771416

I check the job-output on comment #5 and the patch wasn't there yet. This patch was merged on January 29

[Sandeep Yadav (sandeepyadav93)]

Closing this bug as addressed by https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/771416.

If you are still hitting the issue, Please feel free to reopen this bug.