centos-8 container builds fail in promotion on base container

[wes hayutin]

Unable to repro locally, only happens in CI..

02-20 20:56:42 | oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-02-20 20:56:42 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-cxf9jq_0/docker/base/base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-base:a1bd02d30481c514e0a16c6d6ce5ea458f7d1001_750cce02 /tmp/kolla-cxf9jq_0/docker/base
2020-02-20 20:56:42 | Exit code: 1
2020-02-20 20:56:42 | Stdout: ''
2020-02-20 20:56:42 | Stderr: 'error building at STEP "RUN dnf -y install ca-certificates curl findutils iproute iscsi-initiator-utils lvm2 ncurses procps-ng socat sudo tar which python3 util-linux-user cronie crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo rsync iptables-services && dnf clean all && rm -rf /var/cache/dnf": error while running runtime: exit status 1\n'
2020-02-20 20:56:42 | Unexpected error while running command.
2020-02-20 20:56:42 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-cxf9jq_0/docker/base/base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-base:a1bd02d30481c514e0a16c6d6ce5ea458f7d1001_750cce02 /tmp/kolla-cxf9jq_0/docker/base
2020-02-20 20:56:42 | Exit code: 1
2020-02-20 20:56:42 | Stdout: ''
2020-02-20 20:56:42 | Stderr: 'error building at STEP "RUN dnf -y install ca-certificates curl findutils iproute iscsi-initiator-utils lvm2 ncurses procps-ng socat sudo tar which python3 util-linux-user cronie crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo rsync iptables-services && dnf clean all && rm -rf /var/cache/dnf": error while running runtime: exit status 1\n'
2020-02-20 20:56:42 | Building base image with: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-cxf9jq_0/docker/base/base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-base:a1bd02d30481c514e0a16c6d6ce5ea458f7d1001_750cce02 /tmp/kolla-cxf9jq_0/docker/base

https://logserver.rdoproject.org/64/25064/5/check/periodic-tripleo-centos-8-master-containers-build-push/b2f419b/logs/build.log

[wes hayutin (weshayutin)]

It would be very helpful to have the cleanup of the build logs not removed when python-tripleoclient exists.

https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/utils.py#L110

# the following works locally.
 sudo buildah bud --format docker --tls-verify=False --logfile /tmp/wes/docker/base/base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-base:a1bd02d30481c514e0a16c6d6ce5ea458f7d1001_750cce02 docker/base

[Alex Schultz (alex-schultz)]

Side effect of https://review.opendev.org/#/c/671878/44/tripleoclient/v1/container_image.py. Previously the buildah version was never cleaned up by default (this probably wasn't great either)

[wes hayutin (weshayutin)]

STEP 18: RUN dnf -y install ca-certificates curl findutils iproute iscsi-initiator-utils lvm2 ncurses procps-ng socat sudo tar which python3 util-linux-user cronie crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo rsync iptables-services && dnf clean all && rm -rf /var/cache/dnf
CentOS-8 - AppStream 18 MB/s | 6.4 MB 00:00
CentOS-8 - Base 4.7 MB/s | 5.0 MB 00:01
CentOS-8 - Extras 6.2 kB/s | 2.1 kB 00:00
CentOS-8 - HA 345 kB/s | 538 kB 00:01
CentOS-8 - PowerTools 7.2 MB/s | 2.0 MB 00:00
Package ca-certificates-2019.2.32-80.0.el8_1.noarch is already installed.
Package curl-7.61.1-11.el8.x86_64 is already installed.
Package findutils-1:4.6.0-20.el8.x86_64 is already installed.
Package iproute-4.18.0-15.el8.x86_64 is already installed.
Package procps-ng-3.3.15-1.el8.x86_64 is already installed.
Package tar-2:1.30-4.el8.x86_64 is already installed.
No match for argument: crudini
No match for argument: dumb-init
No match for argument: openstack-tripleo-common-container-base
No match for argument: puppet-tripleo
Error: Unable to find a match: crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo

https://logserver.rdoproject.org/64/25064/6/check/periodic-tripleo-centos-8-master-containers-build-push/3550d66/logs/buildah-builds/kolla-zyg2howw/docker/base/base-build.log

[wes hayutin (weshayutin)]

So the dlrn repos are no longer getting on the base container?

[wes hayutin (weshayutin)]

note.. this is working just fine in check atm

http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-8

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/709039

[Marios Andreou (marios-b)]

also seen in check

    * https://4d6da1b390c8df1f8e50-bd895733351047fc01b6164e721bda2c.ssl.cf5.rackcdn.com/708893/1/check/tripleo-build-containers-centos-8/3af86c3/logs/build.log

    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bc2/707338/4/check/tripleo-build-containers-centos-8/bc234e0/logs/build.log

[Marios Andreou (marios-b)]

14:26 < ykarel> https://review.rdoproject.org/r/#/c/25064/
14:29 < ykarel> marios|ruck, fix is just need to remove https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleoci.yaml#L394

[Marios Andreou (marios-b)]

so it isn't clear if the check errors from comment #7 are the same yet ...

14:35 < ykarel> marios|ruck, in check issue is different
14:35 < ykarel> there 121 images build fine
14:35 < ykarel> but in periodic 0

[Marios Andreou (marios-b)]

Remove buildcontainers_rpm_setup_config for centos8 container build https://review.rdoproject.org/r/25182

[Marios Andreou (marios-b)]

still hitting issues with this job.. .last three builds at [1] are failed... 2 of them look like this error

        * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/9da360e/logs/build.log
                * 2020-02-23 16:36:14 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-v9d3m62l/docker/openstack-base/openstack-base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-openstack-base:d15d61afc27948b715fc611aa20636dd /tmp/kolla-v9d3m62l/docker/openstack-base
2020-02-23 16:36:14 | Exit code: 1

        * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/d8340a7/logs/build.log
                * 2020-02-23 08:35:30 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-_jrjyamd/docker/openstack-base/openstack-base-build.log -t trunk.registry.rdoproject.org/tripleomaster/centos-binary-openstack-base:d15d61afc27948b715fc611aa20636dd /tmp/kolla-_jrjyamd/docker/openstack-base
2020-02-23 08:35:30 | Exit code: 1

but we need the patch in https://review.opendev.org/#/c/709039/ so we can get the logs and see what the actual problem is. I think we leave the bug around for now.

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-8-master-containers-build-push

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709039
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=7aeba8f51afbd2593886318af59697c83932b378
Submitter: Zuul
Branch: master

commit 7aeba8f51afbd2593886318af59697c83932b378
Author: Emilien Macchi <email address hidden>
Date: Thu Feb 20 22:41:18 2020 -0500

    Add --work-dir option to container image build command

    Add --work-dir to openstack overcloud container image build command and
    every run will create a unique workspace which where will be stored Kolla
    configs and build logs. Default directory will be in
    /tmp/container-builds. UUIDs are used to identify each time we run the
    command and will be the directory name in the work dir.

    Depends-On: https://review.opendev.org/#/c/709389/
    Related-Bug: #1864108
    Change-Id: Id3e52ba920c54c98529ecb5f723ba452362a0b32

[Marios Andreou (marios-b)]

I think we are done now including the logging improvements green in both periodic and check

        * https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-8-master-containers-build-push
        * http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-8

and we have the buildah logs

        * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/dfce601/logs/buildah-builds/kolla-builds/
        * https://aa4a00d3c35ea084ad41-5c9bf716eadc2e193f03b25f937f429b.ssl.cf2.rackcdn.com/709950/1/check/tripleo-build-containers-centos-8/19f6773/logs/buildah-builds/kolla-builds/

moving fix released move back if you disagree thanks

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/714437

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714437
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=7f4622ddf8dfbf0d6f0319fc6a882c7045a29f8e
Submitter: Zuul
Branch: stable/train

commit 7f4622ddf8dfbf0d6f0319fc6a882c7045a29f8e
Author: Emilien Macchi <email address hidden>
Date: Thu Feb 20 22:41:18 2020 -0500

    Add --work-dir option to container image build command

    Add --work-dir to openstack overcloud container image build command and
    every run will create a unique workspace which where will be stored Kolla
    configs and build logs. Default directory will be in
    /tmp/container-builds. UUIDs are used to identify each time we run the
    command and will be the directory name in the work dir.

    Related-Bug: #1864108
    Change-Id: Id3e52ba920c54c98529ecb5f723ba452362a0b32
    (cherry picked from commit 7aeba8f51afbd2593886318af59697c83932b378)