package install or updates fail in TripleO due to the ssl cert not being mounted in each container

Bug #1854685 reported by chandan kumar on 2019-12-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

While fixing https://bugs.launchpad.net/tripleo/+bug/1854215 ->
libpod rpm does not build on rhel-8/centos-8 missing deps we have updated the new rhui rpms in RHEL-8 nodepool image which includes code ready related repos but right now in RHEL8 master build images
RHUI is failed to sync those repos.

From following logs:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/2b793ee/build.log

and

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/9597653/build.log

2019-12-02 00:24:15.049 | > repo: downloading from remote: delorean
2019-12-02 00:24:15.049 | > delorean-openstack-tripleo-heat-templates-compa 462 kB/s | 1.0 MB 00:02
2019-12-02 00:24:15.049 | > not found other for: delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af
2019-12-02 00:24:15.049 | > not found modules for: delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af
2019-12-02 00:24:15.049 | > not found deltainfo for: delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af
2019-12-02 00:24:15.049 | > not found updateinfo for: delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af
2019-12-02 00:24:15.049 | > delorean: using metadata from Fri Nov 29 08:44:45 2019.
2019-12-02 00:24:15.049 | > repo: downloading from remote: delorean-rhel8-master-deps
2019-12-02 00:24:15.049 | > dlrn-master-rhel8-deps 838 kB/s | 2.4 MB 00:02
2019-12-02 00:24:15.049 | > not found other for: dlrn-master-rhel8-deps
2019-12-02 00:24:15.049 | > not found modules for: dlrn-master-rhel8-deps
2019-12-02 00:24:15.049 | > not found deltainfo for: dlrn-master-rhel8-deps
2019-12-02 00:24:15.049 | > not found updateinfo for: dlrn-master-rhel8-deps
2019-12-02 00:24:15.049 | > delorean-rhel8-master-deps: using metadata from Thu Nov 28 06:32:11 2019.
2019-12-02 00:24:15.049 | > repo: downloading from remote: rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms
2019-12-02 00:24:15.049 | > error: Status code: 403 for https://rhui-cds/pulp/repos/content/dist/rhel8/rhui/8/x86_64/codeready-builder/os/repodata/repomd.xml (https://rhui-cds/pulp/repos/content/dist/rhel8/rhui/8/x86_64/codeready-builder/os/repodata/repomd.xml).
2019-12-02 00:24:15.049 | > Red Hat CodeReady Linux Builder for RHEL 8 x86_ 5.7 kB/s | 286 B 00:00
2019-12-02 00:24:15.049 | > Cannot download 'https://rhui-cds/pulp/mirror//content/dist/rhel8/rhui/8/x86_64/codeready-builder/os': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2019-12-02 00:24:15.049 | > Error: Failed to synchronize cache for repo 'rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms'
2019-12-02 00:24:15.049 | returncode: 1

It needs to be reproduced locally why it started failing.

chandan kumar (chkumar246) wrote :

I tried the rhui repo locally in RHEL8 ubi image, it works fine.
[root@ab87289106ac /]# yum repolist
Last metadata expiration check: 0:00:11 ago on Mon Dec 2 06:19:21 2019.
repo id repo name status
rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs) from RHUI 1827
rhui-custom-deps Custom Repositories - deps 980
rhui-rhel-8-for-x86_64-appstream-rhui-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream from RHUI (RPMs) 8232
rhui-rhel-8-for-x86_64-baseos-rhui-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS from RHUI (RPMs) 3312
rhui-rhel-8-for-x86_64-highavailability-rhui-rpms Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs) from RHUI 138
ubi-8-appstream Red Hat Universal Base Image 8 (RPMs) - AppStream 1322
ubi-8-baseos Red Hat Universal Base Image 8 (RPMs) - BaseOS 661
ubi-8-codeready-builder Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 12
[root@ab87289106ac /]#

It might be related to new cert issue?

tags: added: promotion-blocker
chandan kumar (chkumar246) wrote :

On more debugging:
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ vi build_images.sh
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ yum repolist
delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af 8.0 kB/s | 3.0 kB 00:00
dlrn-master-rhel8-deps 8.4 kB/s | 3.0 kB 00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs) from RHUI 0.0 B/s | 0 B 00:00
Failed to download metadata for repo 'rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms'
[Errno 13] Permission denied: '/var/cache/dnf/expired_repos.json'
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ sudo dnf config-manager --enable rhui-*;
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ yum repolist
delorean-openstack-tripleo-heat-templates-compat-83439525641e9908d75ed8ba8bced96fcef640af 8.1 kB/s | 3.0 kB 00:00
dlrn-master-rhel8-deps 8.3 kB/s | 3.0 kB 00:00
Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs) from RHUI 0.0 B/s | 0 B 00:00
Failed to download metadata for repo 'rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms'
[Errno 13] Permission denied: '/var/cache/dnf/expired_repos.json'
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ sudo dnf clean all;
76 files removed
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ sudo dnf config-manager --enable rhui-*;
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$ yum repolist
Cannot create repo temporary directory "/var/cache/dnf/delorean-a08e4778268a1ec6/tmpdir.XPIyvP": Permission denied
[Errno 13] Permission denied: '/var/cache/dnf/expired_repos.json'
[zuul@upstream-rhel-8-rdo-cloud-tripleo-0002254550 workspace]$

chandan kumar (chkumar246) wrote :
Download full text (6.1 KiB)

I am finally able to reproduce it to rhel8 ubi image which has older version of rhui rpm installed there.

When we try to install new RHUI rpm on RHEL8 rdocloud image.
[cloud-user@rhel8dib ~]$ sudo yum -y install rdo-rhui-2.2-1.noarch.rpm
Last metadata expiration check: 0:02:54 ago on Tue 03 Dec 2019 01:25:50 AM EST.
Dependencies resolved.
==============================================================================================================================================================================
 Package Arch Version Repository Size
==============================================================================================================================================================================
Installing:
 rdo-rhui noarch 2.2-1 @commandline 8.2 k

Transaction Summary
==============================================================================================================================================================================
Install 1 Package

Total size: 8.2 k
Installed size: 10 k
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing : 1/1
  Installing : rdo-rhui-2.2-1.noarch 1/1
  Running scriptlet: rdo-rhui-2.2-1.noarch 1/1
warning: /etc/pki/rhui/key.pem saved as /etc/pki/rhui/key.pem.rpmorig
warning: /etc/pki/rhui/product/content.crt saved as /etc/pki/rhui/product/content.crt.rpmorig
warning: /etc/yum.repos.d/rh-cloud.repo saved as /etc/yum.repos.d/rh-cloud.repo.rpmorig

  Verifying : rdo-rhui-2.2-1.noarch 1/1

Installed:
  rdo-rhui-2.2-1.noarch

Complete!

And when we do on normal ubi image then

[root@c0ce26e339f0 /]# yum install http://file.rdu.redhat.com/~apevec/OSP/rdo-rhui-2.2-1.noarch.rpm
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Red Hat Universal Base Image 8 (RPMs) - BaseOS 356 kB/s | 759 kB 00:02
Red Hat Universal Base Image 8 (RPMs) - AppStream ...

Read more...

chandan kumar (chkumar246) wrote :

Sorry for the too much noise, As per this https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleoci.yaml#L49

we need to update the new RHUI rpm on rcm server.

Marios Andreou (marios-b) wrote :

ack thanks @chandan as per irc i uploaded the latest rhui rpm to the rcm guest few mins ago:

scp rdo-rhui-2.2-1.noarch.rpm centos@(redacted):/var/www/rcm-guest/rhui/
rdo-rhui-2.2-1.noarch.rpm 100% 8355 1.9MB/s 00:00

        * -rw-rw-r--. 1 centos centos 8355 Dec 3 09:38 rdo-rhui-2.2-1.noarch.rpm

Marios Andreou (marios-b) wrote :

trying to get clarification in sf-ops too but my plan is to wget the rhel-8 guest from rcm server use virt-customize to install latest rhui and re-upload it. I'll use a new filename and not replace the existing image yet.

chandan kumar (chkumar246) wrote :

For RHEL8 container build, we pushed the containers manually on rdo registry to unblock rhel8 master check jobs but now the same issue started appearing during standlaone deploy also.

http://logs.rdoproject.org/82/696182/4/openstack-check/tripleo-ci-rhel-8-scenario001-standalone-rdo/81fe7f9/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

error running [bash -x /tmp/yum_update.sh gating-repo,delorean-current] in container \"rhel-binary-redis-working-container\": error while running runtime: exit status 1\"\\n error while running runtime: exit status 1\\n stderr_lines: <omitted>\\n stdout: \\'Red Hat CodeReady Linux Builder for RHEL 8 x86_

We might need to update this container https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/templates/Dockerfile.j2#L1
trunk.registry.rdoproject.org/rhel/rhel8-rhui:8.0-126 to include latest rhui image

Marios Andreou (marios-b) wrote :

I had to use my laptop as i hit https://access.redhat.com/solutions/4073061 trying with rdo centos7 vm

It finally finished uploading :D ... I attach also the full virt-customize log as I saw some 'error:' in there though it appeared to finish correctly, capturing details here for posterity/checking if needed.

I used a different filename and we can rename later once we're ready. The updated file is called "rhel-8.0-guest-rhui.qcow2.UPDATED-rdo-rhui-2.2-1"

        * [centos@rcn-share rhel-8.0-base]$ pwd
/var/www/rcm-guest/images/rhel_base/rhel-8.0-base

        * -rwxrwxr-x. 1 centos centos 776208384 Aug 5 17:29 rhel-8.0-guest-rhui.qcow2
-rw-r--r--. 1 centos centos 698178586 Aug 5 17:30 rhel-8.0-guest-rhui.qcow2.tar
-rw-rw-r--. 1 centos centos 814743552 Dec 3 11:53 rhel-8.0-guest-rhui.qcow2.UPDATED-rdo-rhui-2.2-1

Marios Andreou (marios-b) wrote :

as per call just now @chkumar|ruck I renamed the file so its ready to go:

        * [centos@rcn-share rhel-8.0-base]$ mv rhel-8.0-guest-rhui.qcow2 rhel-8.0-guest-rhui.qcow2.backup
            [centos@rcn-share rhel-8.0-base]$ mv rhel-8.0-guest-rhui.qcow2.UPDATED-rdo-rhui-2.2-1 rhel-8.0-guest-rhui.qcow2

        * [centos@rcn-share rhel-8.0-base]$ chmod 775 rhel-8.0-guest-rhui.qcow2
            [centos@rcn-share rhel-8.0-base]$ ll
            total 2235480
            -rwxrwxr-x. 1 centos centos 814743552 Dec 3 11:53 rhel-8.0-guest-rhui.qcow2
            -rwxrwxr-x. 1 centos centos 776208384 Aug 5 17:29 rhel-8.0-guest-rhui.qcow2.backup
            -rw-r--r--. 1 centos centos 698178586 Aug 5 17:30 rhel-8.0-guest-rhui.qcow2.tar

wes hayutin (weshayutin) wrote :

FAILED! => changed=true \n cmd:\n - buildah\n - run\n - --volume\n - /tmp/ansible.8ex_x04j:/tmp/yum_update.sh\n - --volume\n - /etc/yum.repos.d:/etc/yum.repos.d\n - --volume\n - /opt/gating_repo:/opt/gating_repo\n - --user\n - root\n - --net\n - host\n - rhel-binary-redis-working-container\n - /tmp/yum_update.sh\n - gating-repo,delorean-current\n delta: \'0:00:06.241846\'\n end: \'2019-12-03 07:42:05.193798\'\n msg: non-zero return code\n rc: 1\n start: \'2019-12-03 07:41:58.951952\'\n stderr: |-\n Failed to download metadata for repo \'rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms\'\n

The issue here is that the containers are not getting the updated ssl certs on the bm node in /etc/pki. TripleO needs to mount /etc/pki from the host to the containers during tripleo-modify-image and deployment.

This would break a customer using RHSM for example.

Changed in tripleo:
status: Confirmed → Triaged
chandan kumar (chkumar246) wrote :

@marios, rekicked the build overcloud image job here https://review.rdoproject.org/r/#/c/23901/

chandan kumar (chkumar246) wrote :

kicked standalone RHEL8 job https://review.rdoproject.org/r/23921 with tripleo modify image patch https://review.opendev.org/#/c/697074/

Marios Andreou (marios-b) wrote :

OK so for clarity that https://review.opendev.org/#/c/697074/ and the updated base image on the rcm server from comment #9 above were the needed things.

chkumar|ruck and marios|rover agree we should close this now.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments