centos-7 overcloud image build jobs are failing on pre-install.d/03-centos-rt - rt repo no longer exists

[Ronelle Landy]

The centos 7 overcloud image jobs are failing on the step: "dib-run-parts Running /tmp/in_target.d/pre-install.d/03-centos-rt" as the rt repo no longer exists in centos 7: http://mirror.centos.org/centos-7/7/

2020-11-16 23:20:26.551 | dib-run-parts Running /tmp/in_target.d/pre-install.d/03-centos-rt
2020-11-16 23:20:26.553 | + '[' 1 -gt 0 ']'
2020-11-16 23:20:26.554 | + set -x
2020-11-16 23:20:26.554 | + set -eu
2020-11-16 23:20:26.554 | + set -o pipefail
2020-11-16 23:20:26.554 | + [[ centos7 == *\c\e\n\t\o\s\7* ]]
2020-11-16 23:20:26.555 | ++ yum repolist all
2020-11-16 23:20:26.556 | ++ grep centos7-rt
2020-11-16 23:20:26.556 | ++ awk '{print $1}'
2020-11-16 23:20:26.556 | ++ head -n 1
2020-11-16 23:20:30.198 | ++ true
2020-11-16 23:20:30.198 | + current_rt_repo=
2020-11-16 23:20:30.198 | + [[ '' == '' ]]
2020-11-16 23:20:30.198 | + cat
2020-11-16 23:20:30.203 | dib-run-parts 03-centos-rt completed

....

2020-11-16 23:20:30.948 | >
2020-11-16 23:20:30.948 | > yum-config-manager --save --setopt=centos7-rt.skip_if_unavailable=true
2020-11-16 23:20:30.949 | >
2020-11-16 23:20:30.949 | > failure: repodata/repomd.xml from centos7-rt: [Errno 256] No more mirrors to try.
2020-11-16 23:20:30.949 | > http://mirror.centos.org/centos/7/rt/x86_64/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
2020-11-16 23:20:30.949 | returncode: 1

The full log is included below:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_376/761892/6/check/tripleo-buildimage-overcloud-hardened-full-centos-7/3766f06/build.log

https://opendev.org/openstack/tripleo-puppet-elements/src/branch/master/elements/overcloud-compute/pre-install.d/03-centos-rt is no longer needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/762935

[Marios Andreou (marios-b)]

duplicate for https://bugs.launchpad.net/tripleo/+bug/1904418 ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762935
Committed: https://git.openstack.org/cgit/openstack/tripleo-puppet-elements/commit/?id=8a2fa38ec16fd041812fc84a8df95ae7ce9a70d3
Submitter: Zuul
Branch: stable/queens

commit 8a2fa38ec16fd041812fc84a8df95ae7ce9a70d3
Author: Ronelle Landy <email address hidden>
Date: Mon Nov 16 20:14:42 2020 -0500

    Remove 03-centos-rt element - the rt repo no longer exists

    http://mirror.centos.org/centos-7/7/ no longer
    has an rt repo. This element is causing image
    builds to fail on all centos-7 platforms.

    Change-Id: I7f492d314cf452daaebaeb1d1cf4383511588964
    Closes-Bug: #1904483

[Marios Andreou (marios-b)]

another related fix there

 Remove rt repo that no longer exists in centos 7 https://review.opendev.org/#/c/762926

[Marios Andreou (marios-b)]

From the notes @ https://hackmd.io/R0kCgz_7SHSix_cNgoC9pg?both i checked and indeed this is also on train https://opendev.org/openstack/tripleo-puppet-elements/src/commit/03fafeb73fd4b6f2f9019c87fc87c559285fb8fd/elements/overcloud-compute/pre-install.d/03-centos-rt so really it should have been posted there first then cherrypicked back to stein/rocky and finally queens (i.e. https://review.opendev.org/762935)

However, I had a chat with ykarel on irc this morning, apparently the rt repo is being re-added so we may want to revert this once it is available...

So let's discuss this afternoon? If we don't *need* to remove that i.e. if it is now working again, then let's not also post to train/stein/rocky and we might even revert the queens one https://review.opendev.org/762935

10:57 < ykarel> marios|ruck, don't know if this is the right fixes
10:58 < marios|ruck> ykarel: k please comment somewhere ideally on one of those bugs?
10:58 < ykarel> if repo is removed, it should get re added
10:58 < ykarel> i am following up with centos team
10:59 < ykarel> if the plan is to stop testing rt repos at all then it's fine but not just due to repo missing
11:00 < ykarel> as per centos team they have added it and should be available in mirrors in some time
11:00 < ykarel> H<HArrfabH>H ykarel: content back in place so fetched through mirror.centos.org pool in the next minutes
11:00 < marios|ruck> ykarel: i suspect changing the mirrors with
                     https://review.opendev.org/#/c/762812/1/playbooks/roles/mirror-update/files/centos-mirror-update may mean we get
                     fresh content which includes the rt repo?
11:01 < ykarel> marios|ruck, it's there now http://mirror.centos.org/centos/7/rt/
11:01 < ykarel> infra mirrors should pick it to in some time
11:01 < marios|ruck> ykarel: yeah here it is 404 for me still
11:02 < marios|ruck> 11:01 < ykarel> marios|ruck, it's there now http://mirror.centos.org/centos/7/rt/
11:02 < ykarel> marios|ruck, wrt https://review.opendev.org/#/c/762812/1/playbooks/roles/mirror-update/files/centos-mirror-update this
                will work only when //mirrors.kernel.org/centos get's updated
11:02 < ykarel> marios|ruck, yes it takes some time to propogate all mirrors
11:03 < ykarel> earlier one looks fine now
11:03 < ykarel> http://mirror.lstn.net/centos/7/rt/

[Marios Andreou (marios-b)]

so as just discussed on the tripleo-ci community call, the rt repos *are* now available e.g. http://mirror.centos.org/centos/7/rt/x86_64/repodata/repomd.xml

not sure if that is some combination of jpena patch https://review.opendev.org/#/c/762812 fetching new repo content or per ykarel discussions today the rt repos were removed for _reasons_ but they are now back again

Our decision now is to either revert https://review.opendev.org/#/c/762926 & https://review.opendev.org/762935
OR
put https://review.opendev.org/762935 on train/rocky/stein as well...

[yatin (yatinkarel)]

Followed up with #centos-devel wrt missing rt repo in 7.9 updates, now rt repo is added so workarounds can be avoided if jobs are passing already:-

Nov 17 14:21:13 <ykarel_> Hi rt repo missing in 7.9, is this a known thing?
Nov 17 14:21:29 <ykarel> in 7.8 it was there https://vault.centos.org/7.8.2003/rt/x86_64/
Nov 17 14:21:51 <ykarel> in 7.9 it's missing http://mirror.centos.org/centos/7.9.2009/rt
Nov 17 14:22:15 <Arrfab> ykarel: worth asking hughesjr ? I can restore from vault if needed
Nov 17 14:22:31 <Arrfab> but probably of the repos he forgot to copy/move for the release
Nov 17 14:22:47 <Arrfab> is that still a valid repo or who's maintaining it ? (which SIG)
Nov 17 14:23:21 <ykarel> Arrfab, /me not aware who maintains that, need to file some ticket to get it available for centos7.9?
Nov 17 14:23:35 <ykarel> we have some jobs which are failing due to missing that repo
Nov 17 14:23:44 <ykarel> https://bugs.launchpad.net/tripleo/+bug/1904483
Nov 17 14:24:55 <Arrfab> ykarel: fwiw, it seems that it's not maintained (by looking at the kernel NVR)
Nov 17 14:26:25 <Arrfab> ykarel: content back in place so fetched through mirror.centos.org pool in the next minutes
Nov 17 14:30:58 <ykarel> Arrfab, Thanks will keep an eye on mirror.centos
Nov 17 14:31:12 <ykarel> ok i see it now
Nov 17 14:31:13 <ykarel> http://mirror.centos.org/centos/7/rt

Nov 17 14:48:46 <TrevorH> Arrfab: kernel-rt is a hughesjr maintained thing but only when someone pokes him to remember to do it
Nov 17 14:50:31 <Arrfab> TrevorH: hmm, ok

Nov 17 17:41:53 <hughesjr> ykarel: i am building the latest one (from 11/10) now (kernel-rt)
Nov 17 17:49:27 <ykarel> hughesjr, ack Thanks, btw where it get's build? i didn't find it in CBS https://cbs.centos.org/koji/index
Nov 17 17:50:06 <hughesjr> ykarel: in the c7 os build system
Nov 17 17:50:16 <hughesjr> BUT
Nov 17 17:50:39 <hughesjr> kernel-rt specifically .. on the secureboot builder which is a manual process
Nov 17 17:51:07 <ykarel> hughesjr, ohkk
Nov 17 17:51:58 <hughesjr> (the other pieces (rtsetup, etc)output here: https://buildlogs.centos.org/c7-rt/
Nov 17 17:52:20 <ykarel> hughesjr, btw Arrfab has already pushed 7.8 builds in 7.9, i guess that not compatible with 7.9 kernels?
Nov 17 17:52:30 <hughesjr> I also copy the kernel into there when competed on the secureboot builder
Nov 17 17:52:34 <ykarel> so those needs to be removed? or is it ok to keep them there only
Nov 17 17:53:00 <hughesjr> i will keep a couple there and put in the new one
Nov 17 17:53:07 <ykarel> okk
Nov 17 17:53:15 <hughesjr> asumming it builds :)

[Marios Andreou (marios-b)]

the change at https://review.opendev.org/#/c/762935/ was reverted with https://review.opendev.org/#/c/763014/

I think we can close this out now moving fix-released please move it back and shout if you disagree thank you

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-puppet-elements queens-eol release.