overcloud deploy failing on fs030 and fs016 while pulling mariadb container from undercloud registry

[chandan kumar]

The tripleo ci team is working on adding fs030 job to periodic pipeline and tobiko team is working on consuming fs016 for running multinode job upstream.

Fs030: https://review.rdoproject.org/r/25550
fs016: https://review.opendev.org/683126

Both both the jobs, overcloud deployment is failing while pulling containers from undercloud registry, Below is the logs:
https://55d217e0e7dc5c9a38a1-bf339ca8b211499ccf5ec38f530d7d51.ssl.cf1.rackcdn.com/683126/31/check/tobiko-tripleo-multinode-containers/8eb4c4c/logs/undercloud/home/zuul/overcloud_deploy.log

2020-03-13 15:31:40 |
2020-03-13 15:31:40 | TASK [tripleo_container_tag : Pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128 image] ***
2020-03-13 15:31:40 | task path: /usr/share/ansible/roles/tripleo_container_tag/tasks/main.yml:27
2020-03-13 15:31:40 | Friday 13 March 2020 15:31:40 +0000 (0:00:00.208) 0:15:34.881 **********
2020-03-13 15:31:40 | fatal: [centos-8-rax-dfw-0015201030]: FAILED! => changed=true
2020-03-13 15:31:40 | cmd: podman pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128
2020-03-13 15:31:40 | delta: '0:00:00.346553'
2020-03-13 15:31:40 | end: '2020-03-13 15:31:40.909830'
2020-03-13 15:31:40 | msg: non-zero return code
2020-03-13 15:31:40 | rc: 125
2020-03-13 15:31:40 | start: '2020-03-13 15:31:40.563277'
2020-03-13 15:31:40 | stderr: |-
2020-03-13 15:31:40 | Trying to pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128...
2020-03-13 15:31:40 | Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client
2020-03-13 15:31:40 | Error: error pulling image "192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128": unable to pull 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128: unable to pull image: Error initializing source docker://192.168.24.1:8787/tripleomaster/centos-binary-mariadb:3621159be13b41f8ead2e873b357f4a5-updated-20200313141128: error pinging docker registry 192.168.24.1:8787: Get https://192.168.24.1:8787/v2/: http: server gave HTTP response to HTTPS client
2020-03-13 15:31:40 | stderr_lines: <omitted>
2020-03-13 15:31:40 | stdout: ''
2020-03-13 15:31:40 | stdout_lines: <omitted>

The container is getting pulled from rdo or docker.io registry during tripleo-container-image-prepare.log.
https://logserver.rdoproject.org/50/25550/8/check/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master/f75e2e4/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

ContainerClustercheckImage: 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:c4734267e31fe6dbd01ada7cf568deba-updated-20200316064347

It might be something related to undercloud registry. It needs to be investigated.

[chandan kumar (chkumar246)]

https://logserver.rdoproject.org/50/25550/8/check/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master/f75e2e4/logs/subnode-1/etc/containers/registries.conf.txt.gz
# Registries that do not use TLS when pulling images or uses self-signed
# certificates.
[registries.insecure]
registries = []

registries.insecure should not be empty

[wes hayutin (weshayutin)]

The proper configuration should be the following:
master upstream jobs have this ini config

https://476c17f101a404ac6d5c-d16147cbae4a8d3986d8e1c6485fafc9.ssl.cf2.rackcdn.com/712345/2/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/ef26f34/logs/undercloud/etc/containers/registries.conf
[registries.insecure]
registries = ['undercloud.ctlplane.localdomain', '192.168.24.1', '192.168.24.3']

This job will most likely start failing upstream as well now on the same issue. We just promoted today and this job was not yet in criteria. Should make it easier to fix tbh.

[wes hayutin (weshayutin)]

It's also non-voting http://zuul.openstack.org/job/tripleo-ci-centos-7-scenario007-multinode-oooq-container

[wes hayutin (weshayutin)]

code is here:
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/tasks/tripleo_podman_install.yml#L50

[wes hayutin (weshayutin)]

FIX is here:
https://review.opendev.org/#/c/712013/6/config/general_config/featureset030.yml

[Ronelle Landy (rlandy)]

Fix is in - closing this out

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/713397
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4178d7afcbe11adb2d6f83ed88b039007f256fb8
Submitter: Zuul
Branch: master

commit 4178d7afcbe11adb2d6f83ed88b039007f256fb8
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Mar 17 15:48:07 2020 +0530

    Add OS::TripleO::Services::Podman to sc 1 and sc 7

    OS::TripleO::Services::Podman is needed for scenario 1 and 7
    multinode deployment to set podman related vars during overcloud
    deployment.

    Closes-Bug: #1867599

    Change-Id: Ica6e3b873466852a37e4f4fa135969881f37dba8
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.2.0 release.