tripleo-ci-centos-9-undercloud-upgrade is failing undercloud upgrade - No such file or directory: '/home/zuul/tripleo-deploy/undercloud/tripleo-heat-installer-templates/environments/deployed-server-noop-ctlplane.yaml'

[Ronelle Landy]

tripleo-ci-centos-9-undercloud-upgrade is failing the undercloud upgrade step with:

2022-11-17 16:31:11 | Exception: <urlopen error [Errno 2] No such file or directory: '/home/zuul/tripleo-deploy/undercloud/tripleo-heat-installer-templates/environments/deployed-server-noop-ctlplane.yaml'>
2022-11-17 16:31:11 | Traceback (most recent call last):
2022-11-17 16:31:11 | File "/usr/lib64/python3.9/urllib/request.py", line 1503, in open_local_file
2022-11-17 16:31:11 | stats = os.stat(localfile)
2022-11-17 16:31:11 | FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/tripleo-deploy/undercloud/tripleo-heat-installer-templates/environments/deployed-server-noop-ctlplane.yaml'

The failure started on 11/17:
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

Example logs:

https://52850518c18ef65996ff-4b7d4da09de7dae33770cacbfd550cfb.ssl.cf2.rackcdn.com/864387/2/check/tripleo-ci-centos-9-undercloud-upgrade/871280a/logs/undercloud/home/zuul/undercloud_upgrade.log

https://14928b1c83cb753e9778-e66afa9c5b07ebbf36793ab7ddc56f1f.ssl.cf1.rackcdn.com/864892/1/check/tripleo-ci-centos-9-undercloud-upgrade/fc18059/logs/undercloud/home/zuul/undercloud_upgrade.log

[Ronelle Landy (rlandy)]

Possible patch impact:
https://github.com/openstack/tripleo-heat-templates/commit/98e7c16eff8888672ce84c6ee2af1ce93d0b8b14

[Ronelle Landy (rlandy)]

Testing revert: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/864700

[Ronelle Landy (rlandy)]

Revert worked: https://review.rdoproject.org/zuul/build/998162afa24746cf8f2c6f9595136fe3

[Sandeep Yadav (sandeepyadav93)]

Job is back to green, we don't need this revert anymore:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

We need to check for root cause why job failed earlier.

[Douglas Viroel (dviroel)]

This issue is back, affecting the same job on check/gate:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0

[chandan kumar (chkumar246)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-master&skip=0 is also started failing with same error.

[Takashi Kajinami (kajinamit)]

Please check whether https://review.opendev.org/c/openstack/python-tripleoclient/+/861548 is included in the tested contents properly.
IMO this is not a code bug but something quite wrong with CI. The failure is intermittent and inconsistent.

[Takashi Kajinami (kajinamit)]

I was trying to check the package used in the latest failure[1] but the package used in the job run no longer exists in the mirror.

However, the current package available[2] is a bit older one but contains the above patch which removed usage of the file.

[1]
https://zuul.opendev.org/t/openstack/build/92cff9c1a02e42ebbc220cd1123d8617
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_92c/867189/1/check/tripleo-ci-centos-9-undercloud-upgrade/92cff9c/logs/undercloud/var/log/extra/package-list-installed.txt

python3-tripleoclient.noarch 19.0.1-0.20221209142914.1a5eab6.el9 @delorean-current

[2]
python3-tripleoclient-19.0.1-0.20221208191412.b5e59b5.el9.noarch.rpm

[chandan kumar (chkumar246)]

With this patch 867195: [DNM] Debug undercloud-upgrade | https://review.opendev.org/c/openstack/python-tripleoclient/+/867195, it will give the rpm version of packages needs to be updated.

[Takashi Kajinami (kajinamit)]

Dumping the related conversation in #tripleo

<ykarel> tkajinam, marios|ruck if it's in upgrade job it might be related to versioning thing and iirc it happens after every release until new tags are pushed for master
<ykarel> may be you to recall something with Sem-Ver: api-break
<ykarel> how it was handled in past
<marios|ruck> ykarel: vaguely :)
<marios|ruck> ykarel: but it seems to be fixed now i mean it may be something else
<ykarel> marios|ruck, i think it will be random as dlrn will build package with timestamp so it will depend on when zed/master package last built
<ykarel> https://trunk.rdoproject.org/api-centos9-zed/api/report.html?package=tripleo-ansible vs https://trunk.rdoproject.org/api-centos9-master-uc/api/report.html?package=tripleo-ansible
<marios|ruck> ykarel: ack ok will revisit this in a bit thanks
<ykarel> ack

[Takashi Kajinami (kajinamit)]

OK now I understand what is happening
Currently we have
  19.0.1-0.20221209142914.1a5eab6.el9 in zed
and
  19.0.1-0.20221208191412.b5e59b5.el9 in master.

Because we use the same version for zed and master, and the master package was built BEFORE zed package was build, the package is not updated properly during upgrade.

[Marios Andreou (marios-b)]

yes thanks Takashi for digging and Chandan for posting https://review.opendev.org/c/openstack/releases/+/867196

I spoke with ykarel last Thursday about this (and it has come up before though I was unable to find the launchpad bug #). I was going to work on it on Friday but a personal issue came up and I was out.

We may get some pushback from the release team cores for releases/+/867196 because of the breaking semver (we shouldn't really bump major at this point but we need it to signal the new series) but in the end we should be OK especially now as 'independent' release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/867224

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/867224
Committed: https://opendev.org/openstack/tripleo-ci/commit/44dc3e18f09559dcfc9a350245f0849df26d2eeb
Submitter: "Zuul (22348)"
Branch: master

commit 44dc3e18f09559dcfc9a350245f0849df26d2eeb
Author: Douglas Viroel <email address hidden>
Date: Mon Dec 12 10:54:35 2022 -0300

    Set tripleo-ci-centos-9-undercloud-upgrade to non-voting

    This patch changes tripleo-ci-centos-9-undercloud-upgrade to a
    non-voting job due to an issue, reported in LP 1996933.
    Job can be reverted to voting when the related bug gets fixed.

    Related-Bug: #1996933
    Change-Id: I603875c5633d122d95ea1fed5061eead04e3b0c0

[chandan kumar (chkumar246)]

https://review.opendev.org/c/openstack/releases/+/867196 and https://review.opendev.org/c/openstack/tripleo-ci/+/867291 got merged,

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-master is back to green now. We can move this bug to fix released.