overcloud deployment is not respecting the "DockerInsecureRegistryAddress" address variable for the container registry.

[Pooja Jadhav]

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master consistently failing with below traceback :

2021-04-30 02:47:07 | 2021-04-30 02:47:07.058790 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-neutron-metadata-agent-ovn:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-neutron-metadata-agent-ovn:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-neutron-metadata-agent-ovn:current-tripleo"}
2021-04-30 02:47:07 | 2021-04-30 02:47:07.060762 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:37.323758 | 1.44s
2021-04-30 02:47:07 | 2021-04-30 02:47:07.684681 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-nova-compute:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-nova-compute:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-nova-compute:current-tripleo"}
2021-04-30 02:47:07 | 2021-04-30 02:47:07.686321 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:37.949318 | 2.06s
2021-04-30 02:47:08 | 2021-04-30 02:47:08.408646 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-iscsid:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-iscsid:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-iscsid:current-tripleo"}
2021-04-30 02:47:08 | 2021-04-30 02:47:08.410659 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:38.673653 | 2.79s
2021-04-30 02:47:09 | 2021-04-30 02:47:09.069572 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-nova-libvirt:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-nova-libvirt:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-nova-libvirt:current-tripleo"}
2021-04-30 02:47:09 | 2021-04-30 02:47:09.071138 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:39.334137 | 3.45s
2021-04-30 02:47:09 | 2021-04-30 02:47:09.725411 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-ovn-controller:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-ovn-controller:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-ovn-controller:current-tripleo"}
2021-04-30 02:47:09 | 2021-04-30 02:47:09.726906 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:39.989904 | 4.10s
2021-04-30 02:47:10 | 2021-04-30 02:47:10.295913 | fa163e4a-b25a-f6ff-9913-00000000380d | FATAL | Pre-fetch all the containers | overcloud-novacompute-0 | item=quay.io/tripleomaster/openstack-cron:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg": "Failed to pull image quay.io/tripleomaster/openstack-cron:current-tripleo", "prefetch_image": "quay.io/tripleomaster/openstack-cron:current-tripleo"}
2021-04-30 02:47:10 | 2021-04-30 02:47:10.297122 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:40.560124 | 4.67s
2021-04-30 02:47:10 | 2021-04-30 02:47:10.302826 | fa163e4a-b25a-f6ff-9913-00000000380d | TIMING | Pre-fetch all the containers | overcloud-novacompute-0 | 0:13:40.565830 | 4.68s

References :

https://review.rdoproject.org/zuul/build/19dac6d08da94ef4b4fae0b69708f888/log/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

https://review.rdoproject.org/zuul/build/59ae0a2a88af49f9aaa191ebf7e65a4e/log/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[wes hayutin (weshayutin)]

The deployment should NOT be trying to hit quay which is default in tripleo-common.

openstack overcloud deploy --stack overcloud --override-ansible-cfg /home/zuul/custom_ansible.cfg \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --libvirt-type qemu --timeout 120 --ntp-server 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org -e /home/zuul/cloud-names.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml -e /home/zuul/overcloud-baremetal-deployed.yaml --deployed-server -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /home/zuul/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/network/multiple-nics-ipv6/network-environment.yaml -e /home/zuul/network-environment.yaml -e /home/zuul/overcloud_network_params.yaml -e /home/zuul/overcloud_storage_params.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/zuul/src/opendev.org/openstack/tripleo-ci/test-environments/worker-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/debug.yaml -e /home/zuul/enable-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml -e /home/zuul/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml --validation-warnings-fatal -e /home/zuul/overcloud-topology-config.yaml -e /home/zuul/overcloud-selinux-config.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/ovb-ha.yaml --disable-validations \
    "$@" && status_code=0 || status_code=$?

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/15d20dd/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz

It should hit the undercloud registry as defined here:

parameter_defaults:
  DockerInsecureRegistryAddress:
  - 192.168.24.1:8787
  - docker.io

[wes hayutin (weshayutin)]

registries.conf appears to be configured correctly..

# registries configuration

# insecure registry list
[[registry]]
prefix = "192.168.24.1:8787"
insecure = true
location = "192.168.24.1:8787"
[[registry]]
prefix = "docker.io"
insecure = true
location = "docker.io"

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/15d20dd/logs/overcloud-controller-0/etc/containers/registries.conf.txt.gz

[wes hayutin (weshayutin)]

podman.x86_64 3.0.1-6.module_el8.5.0+736+58cc1a5a @appstream

This is the same version of podman that is working in fs001 jobs

AND... SHIT.. all the overclouds are pulling directly from a registry and not the undercloud registry.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/2cf353a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2021-04-30 11:38:07 | 2021-04-30 11:38:07.818485 | fa163e64-28f7-0f08-c158-000000003637 | CHANGED | Pull quay.io/tripleomaster/openstack-cinder-volume:current-tripleo image | overcloud-controller-0
2021-04-30 11:38:07 | 2021-04-30 11:38:07.820017 | fa163e64-28f7-0f08-c158-000000003637 | TIMING | tripleo_container_tag : Pull quay.io/tripleomaster/openstack-cinder-volume:current-tripleo image | overcloud-controller-0 | 0:15:42.279298 | 107.41s

[wes hayutin (weshayutin)]

The overcloud deployment SHOULD NOT BE PULLING DIRECTLY FROM A REGISTRY :(

[wes hayutin (weshayutin)]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/19dac6d/logs/undercloud/home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/user-environments/tripleoclient-parameters.yaml.txt.gz

DockerInsecureRegistryAddress is not found in the client params

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/789131

[Rabi Mishra (rabi)]

Looks like a regression from https://review.opendev.org/c/openstack/python-tripleoclient/+/769984

https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 should possibly fix it.

[wes hayutin (weshayutin)]

scope of the fall out:
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&branch=master

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/789131
Committed: https://opendev.org/openstack/python-tripleoclient/commit/e36e9562e3e03e078efd545ae55af175ec978cab
Submitter: "Zuul (22348)"
Branch: master

commit e36e9562e3e03e078efd545ae55af175ec978cab
Author: ramishra <email address hidden>
Date: Mon May 3 11:12:55 2021 +0530

    Use environment files when building image params

    Also moves deployed-server-environment.yaml before
    the user environments as it resets
    OS::TripleO::DeployedServer::ControlPlanePort.

    Regression from I8fb6ca088b1052488ff4f9ada4d3ab29c0be4735

    Closes-Bug: #1926758
    Change-Id: I1a6da2e92cc994240f53e8d702f61d941b6a37f7

[Michele Baldessari (michele)]

Is this really fixed? I have got I1a6da2e92cc994240f53e8d702f61d941b6a37f7 in my env (python3-tripleoclient-16.0.1-0.20210504140908.67a5659.el8.noarch) and I deploy with the following:
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--stack overcloud \
-r /home/stack/tht/roles_data_spine_leaf.yaml \
-n /home/stack/tht/network_data_spine_leaf.yaml \
-e /home/stack/tht/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/frr.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cinder-volume-active-active.yaml \
-e /home/stack/tht/network-environment.yaml \
-e /home/stack/tht/extraconfigpre_env.yaml \
-e /home/stack/tht/tls-everywhere.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/inject-trust-anchor.yaml \
-e /home/stack/tht/ips-from-pool-all.yaml \
-e /home/stack/overcloud-bm-deployed.yaml \
-e /home/stack/docker-images.yaml \
-e /home/stack/ovn-updated-images.yaml

The last two files have the images pointing to the undercloud:
(undercloud) [stack@undercloud-0 ~]$ grep quay.io docker-images.yaml ovn-updated-images.yaml |wc -l
0

Yet my stack has all images pointing to quay.io:
(undercloud) [stack@undercloud-0 ~]$ openstack stack environment show overcloud -f yaml |grep Image: |grep quay.io | wc -l
59
(undercloud) [stack@undercloud-0 ~]$ openstack stack environment show overcloud -f yaml |grep Image: |grep quay.io | head -n5
  ContainerBarbicanApiImage: quay.io/tripleomaster/openstack-barbican-api:current-tripleo
  ContainerCinderApiImage: quay.io/tripleomaster/openstack-cinder-api:current-tripleo
  ContainerCinderConfigImage: quay.io/tripleomaster/openstack-cinder-api:current-tripleo
  ContainerCinderSchedulerImage: quay.io/tripleomaster/openstack-cinder-scheduler:current-tripleo
  ContainerCinderVolumeImage: quay.io/tripleomaster/openstack-cinder-volume:current-tripleo

This all worked a few days ago

[Michele Baldessari (michele)]

I reverted https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 and things are back to normal for me:
(undercloud) [stack@undercloud-0 ~]$ openstack stack environment show overcloud -f yaml |grep Image:
  ContainerBarbicanApiImage: undercloud-0.ctlplane.bgp.ftw:8787/tripleomaster/openstack-barbican-api:current-tripleo
  ContainerCephDaemonImage: undercloud-0.ctlplane.bgp.ftw:8787/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64
  ContainerCinderApiImage: undercloud-0.ctlplane.bgp.ftw:8787/tripleomaster/openstack-cinder-api:current-tripleo

I.e. no more quay.io

[Rabi Mishra (rabi)]

> The last two files have the images pointing to the undercloud:

If you're using environments with image parameters (already have images with correct tags in local registry) then I think you should use "--disable-container-prepare" and that would work.

> This all worked a few days ago

As you don't have DockerInsecureRegistryAddress in any of your environments, container image prepare is using the default and overriding what you're providing in docker-images.yaml and ovn-updated-images.yaml.

The order of these environments have been changed in https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 (i.e generated environment with image params overriding the ones you provide). We can revert that part if required.

[Michele Baldessari (michele)]

Thanks Rabi!

> If you're using environments with image parameters (already have images with correct tags in local registry) then I think you should use "--disable-container-prepare" and that would work.

I tried adding this option to my deploy:
#!/bin/bash
source /home/stack/stackrc

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--stack overcloud \
--disable-container-prepare \
-r /home/stack/tht/roles_data_spine_leaf.yaml \
-n /home/stack/tht/network_data_spine_leaf.yaml \
-e /home/stack/tht/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/frr.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/cinder-volume-active-active.yaml \
-e /home/stack/tht/network-environment.yaml \
-e /home/stack/tht/extraconfigpre_env.yaml \
-e /home/stack/tht/tls-everywhere.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-internal-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-everywhere-endpoints-dns.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/haproxy-public-tls-certmonger.yaml \
-e /home/stack/inject-trust-anchor.yaml \
-e /home/stack/tht/ips-from-pool-all.yaml \
-e /home/stack/overcloud-bm-deployed.yaml \
-e /home/stack/docker-images.yaml \
-e /home/stack/ovn-updated-images.yaml

But I still get the same issue:
(undercloud) [stack@undercloud-0 tht]$ openstack stack environment show overcloud |grep Image:
  ContainerBarbicanApiImage: quay.io/tripleomaster/openstack-barbican-api:current-tripleo
  ContainerCephDaemonImage: quay.ceph.io/ceph-ci/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64
  ContainerCinderApiImage: quay.io/tripleomaster/openstack-cinder-api:current-tripleo
  ContainerCinderConfigImage: quay.io/tripleomaster/openstack-cinder-api:current-tripleo

> As you don't have DockerInsecureRegistryAddress in any of your environments, container image prepare is using the default and overriding what you're providing in docker-images.yaml and ovn-updated-images.yaml.

> The order of these environments have been changed in https://review.opendev.org/c/openstack/python-tripleoclient/+/789131 (i.e generated environment with image params overriding the ones you provide). We can revert that part if required.

I do have that set though:
(undercloud) [stack@undercloud-0 tht]$ openstack stack environment show overcloud |grep -A2 DockerIns
  DockerInsecureRegistryAddress:
  - undercloud-0.ctlplane.bgp.ftw:8787

My ~/docker-images.yaml is simply the output of
sudo openstack tripleo container image prepare -e /home/stack/containers-prepare-parameter.yaml --output-env-file /home/stack/docker-images.yaml

Are there any other changes/workarounds I could use/apply?

thanks!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/789917

[Rabi Mishra (rabi)]

> But I still get the same issue

I've changed the order of environments with https://review.opendev.org/c/openstack/python-tripleoclient/+/789917 to work as before and address this kind of corner case along with some refactoring. See if it helps.

[Rabi Mishra (rabi)]

> I do have that set though:
>(undercloud) [stack@undercloud-0 tht]$ openstack stack environment show overcloud |grep -A2 >DockerIns
> DockerInsecureRegistryAddress:
> - undercloud-0.ctlplane.bgp.ftw:8787

Actually undercloud registry is not used for image parameters unless you've ContainerImagePrepare parameter in an environment with 'push_destination: true'[1] in spite of undercloud registry being configured.

[1] https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/kolla_builder.py#L382

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/790129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/789917
Committed: https://opendev.org/openstack/python-tripleoclient/commit/0b9e9a43c85d41ffb9f19c1180b607d3d70ad96e
Submitter: "Zuul (22348)"
Branch: master

commit 0b9e9a43c85d41ffb9f19c1180b607d3d70ad96e
Author: ramishra <email address hidden>
Date: Wed May 5 15:25:42 2021 +0530

    Allow user environments override generated ones

    We generate some parameters using the user environments like
    the one for image parameters. However, user environments may
    have parameters to override some of those. Make sure the ones
    in user environments takes precedence by adding the user
    environments at the end.

    Additional changes:

    - Refactors to use the correct image for ephemeral heat
    - Removes ntp server check as we never checked for
      ControllerCount in all environment files and also use
      a hard-coded role name.

    Related-Bug: #1926758
    Change-Id: I6e6ee98cb6a06bd830eb712f67e1cd0d9c13a1f4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/790224

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/790129
Committed: https://opendev.org/openstack/python-tripleoclient/commit/588eece6b579bc0f0ab989aeaae944bd6a9e6262
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 588eece6b579bc0f0ab989aeaae944bd6a9e6262
Author: ramishra <email address hidden>
Date: Mon May 3 11:12:55 2021 +0530

    Use environment files when building image params

    Also moves deployed-server-environment.yaml before
    the user environments as it resets
    OS::TripleO::DeployedServer::ControlPlanePort.

    Regression from I8fb6ca088b1052488ff4f9ada4d3ab29c0be4735

    Closes-Bug: #1926758
    Change-Id: I1a6da2e92cc994240f53e8d702f61d941b6a37f7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/790224
Committed: https://opendev.org/openstack/python-tripleoclient/commit/0c7a109ead3877976ff35a435a89a3f40a4951b4
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0c7a109ead3877976ff35a435a89a3f40a4951b4
Author: ramishra <email address hidden>
Date: Wed May 5 15:25:42 2021 +0530

    Allow user environments override generated ones

    We generate some parameters using the user environments like
    the one for image parameters. However, user environments may
    have parameters to override some of those. Make sure the ones
    in user environments takes precedence by adding the user
    environments at the end.

    Additional changes:

    - Refactors to use the correct image for ephemeral heat
    - Removes ntp server check as we never checked for
      ControllerCount in all environment files and also use
      a hard-coded role name.

    Related-Bug: #1926758
    Change-Id: I6e6ee98cb6a06bd830eb712f67e1cd0d9c13a1f4
    (cherry picked from commit 0b9e9a43c85d41ffb9f19c1180b607d3d70ad96e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 16.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 17.0.0 release.