master & wallaby overcloud periodic jobs pulling heat-api and heat-engine current-tripleo instead of tripleo-ci-testing

[Marios Andreou]

The master and wallaby overcloud periodic jobs (i.e. not standalone/undercloud) are pulling the current-tripleo version of the heat-api and heat-engine containers instead of using the tripleo-ci-testing hash. Examples at [1][2] like:

 2021-06-15 02:11:40 | 2021-06-15 02:11:40.522 112912 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Using ephemeral heat for stack operation
 2021-06-15 02:11:41 | 2021-06-15 02:11:41.447 112912 INFO tripleoclient.heat_launcher [-] Pulling conatiner image quay.io/tripleomaster/openstack-heat-api:current-tripleo.
 2021-06-15 02:11:41 | Trying to pull quay.io/tripleomaster/openstack-heat-api:current-tripleo...
    ...
 2021-06-15 02:12:00 | 2021-06-15 02:12:00.839 112912 INFO tripleoclient.heat_launcher [-] Pulling conatiner image quay.io/tripleomaster/openstack-heat-engine:current-tripleo.

This is specific to master and wallaby and limited to those two containers. Not a blocker but this means the integration line is degraded in particular for testing the heat-engine and heat-api containers, so adding the promotion-blocker tag too.

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/6a112ff/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-wallaby/3946fd7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Marios Andreou (marios-b)]

The relevant code is there [1]

         for image in self.api_container_image, self.engine_container_image:
            log.info("Pulling conatiner image {}.".format(image))

And I believe the problem is that it uses the defaults values for the api_container_image and engine_container_image from [2]

        DEFAULT_CONTAINER_TAG = "current-tripleo"
        DEFAULT_CONTAINER_REGISTRY = "quay.io"
 DEFAULT_HEAT_ENGINE_CONTAINER = ('{}/{}/openstack-heat-engine:{}'.format(
     DEFAULT_CONTAINER_REGISTRY,
     DEFAULT_CONTAINER_NAMESPACE,
     DEFAULT_CONTAINER_TAG))

[1] https://opendev.org/openstack/python-tripleoclient/src/commit/7797524f71fa284db0ae37c0e0f3c4f6ebeebc7c/tripleoclient/heat_launcher.py#L469-L470
[2] https://opendev.org/openstack/python-tripleoclient/src/commit/7797524f71fa284db0ae37c0e0f3c4f6ebeebc7c/tripleoclient/constants.py#L47-L54

[Ronelle Landy (rlandy)]

This is actually blocking osp-17 - as the default reg it tries to pull from requires login:

2021-06-14 19:04:04 | unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/796516

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/796516
Committed: https://opendev.org/openstack/tripleo-common/commit/31468f279470778d16a73b92a644a52fe4e26598
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 31468f279470778d16a73b92a644a52fe4e26598
Author: Alex Schultz <email address hidden>
Date: Tue Jun 15 09:38:04 2021 -0600

    [Wallaby-only] Update default container namespace for wallaby

    After cutting a stable branch, we need to patch it with the correct
    container information for the release.

    Change-Id: I7db4adaf4509cf45545eb16e777e764a2b19df3b
    Closes-Bug: 1931995

[Marios Andreou (marios-b)]

There is some confusion about the fix. The patch at https://review.opendev.org/c/openstack/tripleo-common/+/796516 while completely necessary is not related to this bug.

The patches at https://review.opendev.org/q/topic:%22prep_images_output%22 are addressing this issue. They provide a way to specify the heat containers to pull (instead of defaulting to current-tripleo master as described in this bug).

[Bogdan Dobrelya (bogdando)]

It is as well addressed by https://review.opendev.org/c/openstack/python-tripleoclient/+/796733

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796617
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/afc0d39dc2dd32ba602aaabebe7e398b2d5c82b9
Submitter: "Zuul (22348)"
Branch: master

commit afc0d39dc2dd32ba602aaabebe7e398b2d5c82b9
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 16 11:19:42 2021 +0200

    Add Ephemeral Heat service

    Adds a ephemeral heat service that ensures the containers are fetched on
    the system and tagged specially for usage later with the overcloud
    deployment process. After the service deploys, the following container
    images should be available on the local system.

       localhost/tripleo/openstack-heat-all:ephemeral
       localhost/tripleo/openstack-heat-engine:ephemeral
       localhost/tripleo/openstack-heat-api:ephemeral

    Depends-On: https://review.opendev.org/c/openstack/tripleo-common/+/796614
    Partial-Bug: #1931995

    Change-Id: I923856c83c14eb54073684ace93e9e1e85f53329
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Alex Schultz <email address hidden>
    Signed-off-by: Alex Schultz <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/796733
Committed: https://opendev.org/openstack/python-tripleoclient/commit/268f638f15305d4d59b9a008307611772cce35f1
Submitter: "Zuul (22348)"
Branch: master

commit 268f638f15305d4d59b9a008307611772cce35f1
Author: James Slagle <email address hidden>
Date: Wed Jun 23 07:35:18 2021 -0400

    Default Heat images to container image prepare defaults

    Use the defaults from container image prepare to set the default values
    for the standalone and undercloud heat api and engine images. This will
    allow the default upstream or downstream defaults from the right branch
    to be used.

    Additionally this patch splits out the ephemeral heat containers to a
    special namespace and tag that we use in THT to setup the containers for
    the overcloud deploy command on an undercloud.

    related-bug: 1931995
    Depends-On: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796617
    Change-Id: I55fb9816e2b0a54717938b7a094bde6780811305
    Signed-off-by: James Slagle <email address hidden>
    Signed-off-by: Alex Schultz <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797765

[Marios Andreou (marios-b)]

almost done here I think... the only patch not merged yet that I am aware of is that one https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797522

as far as i can see the job is no longer trying to pull current-tripleo heat-api/engine containers e.g. from today at [1]

"2021-06-28 02:09:03 | 2021-06-28 02:09:03.540 107741 INFO tripleoclient.heat_launcher [-] Skipping container image pull."

[1] https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/6718510/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797765
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/18b544723ab35ce763e67b517af9383bad846657
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 18b544723ab35ce763e67b517af9383bad846657
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Jun 16 11:19:42 2021 +0200

    Add Ephemeral Heat service

    Adds a ephemeral heat service that ensures the containers are fetched on
    the system and tagged specially for usage later with the overcloud
    deployment process. After the service deploys, the following container
    images should be available on the local system.

       localhost/tripleo/openstack-heat-all:ephemeral
       localhost/tripleo/openstack-heat-engine:ephemeral
       localhost/tripleo/openstack-heat-api:ephemeral

    Depends-On: https://review.opendev.org/c/openstack/tripleo-common/+/796614
    Partial-Bug: #1931995

    Change-Id: I923856c83c14eb54073684ace93e9e1e85f53329
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Alex Schultz <email address hidden>
    Signed-off-by: Alex Schultz <email address hidden>
    (cherry picked from commit afc0d39dc2dd32ba602aaabebe7e398b2d5c82b9)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/python-tripleoclient/+/798824

[Marios Andreou (marios-b)]

wallaby patches merged @ https://review.opendev.org/q/topic:prep_images_output-stable%252Fwallaby

I think we also need to cherrypick that one https://review.opendev.org/c/openstack/python-tripleoclient/+/796733 - I had a go (some merge conflicts) and posted https://review.opendev.org/c/openstack/python-tripleoclient/+/798824 just now - but it looks to me that there are some patches missing in wallaby ? (see comment at https://review.opendev.org/c/openstack/python-tripleoclient/+/798824/1#message-dfabfe6f5a86a8039661003ba1f9158c4066d8bf )

[James Slagle (james-slagle)]

Should be fixed now. There was a patch missing:
https://review.opendev.org/c/openstack/python-tripleoclient/+/798836

And another one that was already backported, but not yet merged that needs to be the parent:
https://review.opendev.org/c/openstack/python-tripleoclient/+/796304

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/python-tripleoclient/+/798824
Committed: https://opendev.org/openstack/python-tripleoclient/commit/18ce8b9a09c21b76cb71a343953c361fc54b9206
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 18ce8b9a09c21b76cb71a343953c361fc54b9206
Author: James Slagle <email address hidden>
Date: Wed Jun 23 07:35:18 2021 -0400

    Default Heat images to container image prepare defaults

    Use the defaults from container image prepare to set the default values
    for the standalone and undercloud heat api and engine images. This will
    allow the default upstream or downstream defaults from the right branch
    to be used.

    Additionally this patch splits out the ephemeral heat containers to a
    special namespace and tag that we use in THT to setup the containers for
    the overcloud deploy command on an undercloud.

    related-bug: 1931995
    Depends-On: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796617
    Change-Id: I55fb9816e2b0a54717938b7a094bde6780811305
    Signed-off-by: James Slagle <email address hidden>
    Signed-off-by: Alex Schultz <email address hidden>
    (cherry picked from commit 268f638f15305d4d59b9a008307611772cce35f1)

[wes hayutin (weshayutin)]

all patches merged, jobs called out are passing.

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-master

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 15.3.0 release.