podman-package-rhel-8 job is giving post_failure after RHEL-8.1 movement

[chandan kumar]

podman-package-rhel-8 job is used to run Build rpm of podman against containers/libpod porject.
It has started giving post_failure from multiple days.

https://review.rdoproject.org/zuul/builds?pipeline=github-check&project=containers%2Flibpod&job_name=podman-package-rhel-8

Below is the error:

https://logserver.rdoproject.org/08/5208/1d25cf2e1d4d0ca450e4c72fc88d99b8ba6cf417/github-check/podman-package-rhel-8/a3b816b/job-output.txt

2020-02-14 04:27:34.840913 | test | Package redhat-rpm-config-120-1.el8.noarch is already installed.
2020-02-14 04:27:34.841567 | test | Package rpm-build-4.14.2-25.el8.x86_64 is already installed.
2020-02-14 04:27:34.842161 | test | Package rpmdevtools-8.10-7.el8.noarch is already installed.
2020-02-14 04:27:34.844578 | test | Package python36-devel-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64 is already installed.
2020-02-14 04:27:34.861722 | test | Error: Unable to find a match: device-mapper-devel glibc-static go-compilers-golang-compiler gpgme-devel libassuan-devel
2020-02-14 04:27:34.912378 | test | make: *** [Makefile:626: package] Error 1
2020-02-14 09:27:34.978816 | test | ERROR

Below is the repo enabled on Red Hat Enterprise Linux release 8.1 (Ootpa)
2020-02-14 09:26:43.149389 | TASK [Enable RHUI repository]
2020-02-14 04:26:43.702704 | test | + dnf config-manager --enable 'rhui-*'
2020-02-14 04:26:44.404383 | test | + dnf makecache
2020-02-14 04:26:44.891382 | test | Custom Repositories - deps 46 kB/s | 2.1 kB 00:00
2020-02-14 04:26:44.973415 | test | Red Hat Enterprise Linux 8 for x86_64 - AppStre 41 kB/s | 2.8 kB 00:00
2020-02-14 04:26:45.104956 | test | Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 kB/s | 2.4 kB 00:00
2020-02-14 04:26:45.244439 | test | Red Hat Enterprise Linux 8 for x86_64 - High Av 50 kB/s | 2.4 kB 00:00
2020-02-14 04:26:45.929868 | test | Metadata cache created.
2020-02-14 04:26:45.977473 | test | + dnf repolist
2020-02-14 04:26:47.263629 | test | Last metadata expiration check: 0:00:02 ago on Fri 14 Feb 2020 04:26:45 AM EST.
2020-02-14 04:26:47.847022 | test | repo id repo name status
2020-02-14 04:26:47.847092 | test | rhui-custom-deps Custom Repositories - d 980
2020-02-14 04:26:47.847099 | test | rhui-rhel-8-for-x86_64-appstream-rhui-rpms Red Hat Enterprise Linu 8,589
2020-02-14 04:26:47.847103 | test | rhui-rhel-8-for-x86_64-baseos-rhui-rpms Red Hat Enterprise Linu 3,676
2020-02-14 04:26:47.847108 | test | rhui-rhel-8-for-x86_64-highavailability-rhui-rpms Red Hat Enterprise Linu 154
2020-02-14 09:26:48.250288 | test | ok: Runtime: 0:00:04.209879

The enabled repos are same. But as we moved to RHEL-8.1 one.
From rhui following packages got missing

device-mapper-devel glibc-static go-compilers-golang-compiler gpgme-devel libassuan-devel

There is no changes in the script which builds the RPM.

Please have a look on RHUI side.

[chandan kumar (chkumar246)]

Logs from working job
https://logserver.rdoproject.org/openstack-regular/github.com/containers/libpod/master/podman-package-rhel-8/5665b5b/job-output.txt

2020-02-06 19:42:40.594244 | TASK [Enable RHUI repository]
2020-02-06 14:42:41.094194 | test | + dnf config-manager --enable 'rhui-*'
2020-02-06 14:42:41.810926 | test | + dnf makecache
2020-02-06 14:42:42.273072 | test | Red Hat CodeReady Linux Builder for RHEL 8 x86_ 55 kB/s | 2.8 kB 00:00
2020-02-06 14:42:42.418620 | test | Red Hat CodeReady Linux Builder for RHEL 8 x86_ 20 MB/s | 3.0 MB 00:00
2020-02-06 14:42:43.041042 | test | Custom Repositories - deps 44 kB/s | 2.1 kB 00:00
2020-02-06 14:42:43.099287 | test | Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 kB/s | 2.8 kB 00:00
2020-02-06 14:42:43.368854 | test | Red Hat Enterprise Linux 8 for x86_64 - AppStre 54 MB/s | 14 MB 00:00
2020-02-06 14:42:46.123011 | test | Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 kB/s | 2.4 kB 00:00
2020-02-06 14:42:46.390692 | test | Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 14 MB 00:00
2020-02-06 14:42:48.973111 | test | Red Hat Enterprise Linux 8 for x86_64 - High Av 42 kB/s | 2.4 kB 00:00
2020-02-06 14:42:49.109616 | test | Red Hat Enterprise Linux 8 for x86_64 - High Av 5.7 MB/s | 787 kB 00:00
2020-02-06 14:42:50.712495 | test | Metadata cache created.
2020-02-06 14:42:50.757670 | test | + dnf repolist
2020-02-06 14:42:51.612095 | test | Last metadata expiration check: 0:00:02 ago on Thu 06 Feb 2020 02:42:49 PM EST.
2020-02-06 14:42:52.214199 | test | repo id repo name status
2020-02-06 14:42:52.214239 | test | rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms Red Hat CodeReady Linu 1,858
2020-02-06 14:42:52.214251 | test | rhui-custom-deps Custom Repositories - 980
2020-02-06 14:42:52.214256 | test | rhui-rhel-8-for-x86_64-appstream-rhui-rpms Red Hat Enterprise Lin 8,589
2020-02-06 14:42:52.214263 | test | rhui-rhel-8-for-x86_64-baseos-rhui-rpms Red Hat Enterprise Lin 3,675
2020-02-06 14:42:52.214267 | test | rhui-rhel-8-for-x86_64-highavailability-rhui-rpms Red Hat Enterprise Lin 154
2020-02-06 19:42:52.671826 | test | ok: Runtime: 0:00:11.162267

2020-02-06 14:43:37.794853 | test | + sudo /bin/dnf install -y createrepo device-mapper-devel git glib2-devel glibc-static go-compilers-golang-compiler golang gpgme-devel libassuan-devel libseccomp-devel libselinux-devel make redhat-rpm-config rpm-build rpmdevtools systemd-devel python3-devel python3-varlink
2020-02-06 14:43:38.416222 | test | Last metadata expiration check: 0:00:49 ago on Thu 06 Feb 2020 02:42:49 PM EST.
2020-02-06 14:43:39.010137 | test | Package createrepo_c-0.11.0-3.el8.x86_64 is already installed.
2020-02-06 14:43:39.012204 | test | Package git-2.18.2-1.el8_1.x86_64 is already installed.
2020-02-06 14:43:39.014452 | test | Package golang-1.12.12-4.module+el8.1.0+5317+8a49ffbb.x86_64 is already installed.
2020-02-06 14:43:39.016644 | test | Package libselinux-devel-2.9-2.1.el8.x86_64 is already installed.
2020-02-06 14:43:39.017...

[chandan kumar (chkumar246)]

From post_failure and success jobs, codeready containers got missing

[chandan kumar (chkumar246)]

https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/podman.yaml#L33
label: upstream-rhel-8 node is used.
from test review https://review.rdoproject.org/r/24944 -> https://logserver.rdoproject.org/44/24944/1/check/podman-package-rhel-8/f75d96c/job-output.txt
2020-02-14 06:13:41.267557 | test | + rpm -qa
2020-02-14 06:13:41.267628 | test | + grep rhui
2020-02-14 06:13:41.661513 | test | rdo-rhui-2.1-1.noarch
2020-02-14 11:13:41.859106 | test | ok: Runtime: 0:00:03.891154

[chandan kumar (chkumar246)]

After rhel-8 nodepool image get refreshed with rhui-2.2 code ready repos are available now and the job is working fine now
https://review.rdoproject.org/zuul/builds?job_name=podman-package-rhel-8

[Marios Andreou (marios-b)]

per chkumar comment #4 above moving fix released please move back if you disagree