containers are built with older tripleo-hash after promotions

[chandan kumar]

On this review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/798260/ in gate queue

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing at undercloud install.
while pulling keystone container [1]
```
Error running container image prepare: Not found image: http://188.212.108.112:5001/v2/tripleomaster/openstack-keystone/manifests/52efe0ec63fdc68d0f626c6d535aa4c1
```
and
By looking at job-output.txt [2]
```
export UNDERCLOUD_INSTALL_RELEASE=master
2021-07-07 07:21:36.899490 | primary | ++(/home/zuul/workspace/logs/releases.sh:2): source(): UNDERCLOUD_INSTALL_RELEASE=master
2021-07-07 07:21:36.899505 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): source(): export UNDERCLOUD_INSTALL_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
2021-07-07 07:21:36.899512 | primary | ++(/home/zuul/workspace/logs/releases.sh:3): source(): UNDERCLOUD_INSTALL_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
2021-07-07 07:21:36.899524 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): source(): export UNDERCLOUD_TARGET_RELEASE=master
2021-07-07 07:21:36.899532 | primary | ++(/home/zuul/workspace/logs/releases.sh:4): source(): UNDERCLOUD_TARGET_RELEASE=master
2021-07-07 07:21:36.899549 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): source(): export UNDERCLOUD_TARGET_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
2021-07-07 07:21:36.899556 | primary | ++(/home/zuul/workspace/logs/releases.sh:5): source(): UNDERCLOUD_TARGET_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
2021-07-07 07:21:36.899572 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): source(): export OVERCLOUD_DEPLOY_RELEASE=master
2021-07-07 07:21:36.899579 | primary | ++(/home/zuul/workspace/logs/releases.sh:6): source(): OVERCLOUD_DEPLOY_RELEASE=master
2021-07-07 07:21:36.899596 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): source(): export OVERCLOUD_DEPLOY_HASH=a87dc5d37d8cb11ad81a78074cc48402
2021-07-07 07:21:36.899603 | primary | ++(/home/zuul/workspace/logs/releases.sh:7): source(): OVERCLOUD_DEPLOY_HASH=a87dc5d37d8cb11ad81a78074cc48402
2021-07-07 07:21:36.899618 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): source(): export OVERCLOUD_TARGET_RELEASE=master
2021-07-07 07:21:36.899624 | primary | ++(/home/zuul/workspace/logs/releases.sh:8): source(): OVERCLOUD_TARGET_RELEASE=master
2021-07-07 07:21:36.899641 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): source(): export OVERCLOUD_TARGET_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
2021-07-07 07:21:36.899648 | primary | ++(/home/zuul/workspace/logs/releases.sh:9): source(): OVERCLOUD_TARGET_HASH=52efe0ec63fdc68d0f626c6d535aa4c1
```
52efe0ec63fdc68d0f626c6d535aa4c1 => current-tripleo-hash
and a87dc5d37d8cb11ad81a78074cc48402 => previous-current-tripleo hash

By looking at the content provider job [3]
```
2021-07-07 08:33:37.277561 | primary | 127.0.0.1:5001/tripleomaster/openstack-keystone a87dc5d37d8cb11ad81a78074cc48402 39c557f93b02 4 hours ago 823 MB
```
openstack-keystone container is built with different hash a87dc5d37d8cb11ad81a78074cc48402 (previous-current-tripleo) on content provider job.

but the update job during undercloud install is expecting container with current-tripleo hash.
leading to undercloud failure.

filing this bug to track further.

Links:
[1.] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_624/798260/2/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/6242807/logs/undercloud/home/zuul/undercloud_install.log

[2.] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_624/798260/2/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/6242807/logs/undercloud/home/zuul/repo_setup.log

[3.] https://bf551380c009c1591acf-43d675d9b4d25dd83e8dc401d48f3c75.ssl.cf1.rackcdn.com/798260/2/gate/tripleo-ci-centos-8-content-provider/2f472a6/job-output.txt

[chandan kumar (chkumar246)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates&pipeline=gate by looking at zuul queue, it is turning Red a lot.

[Cédric Jeanneret (cjeanner)]

Also hit this same issue, on the following change: https://review.opendev.org/c/openstack/tripleo-common/+/799643

It failed in Gate, then the recheck failed.

First it was due to:
http://158.69.73.170:5001/v2/tripleomaster/openstack-rsyslog/manifests/6439b21a91a11b464ad5b2cc147e81cd

Next:
http://23.253.174.55:5001/v2/tripleomaster/openstack-cron/manifests/6439b21a91a11b464ad5b2cc147e81cd

So it's not linked to a precise container, though it's more than probably due to the random order tripleo wants to fetch them...

[Cédric Jeanneret (cjeanner)]

This is also hitting stable/train:
https://9ab22cd0477204e227fb-da16bc011ea15809d69b30dee9f534f2.ssl.cf5.rackcdn.com/799593/1/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train/7d9c9ec/logs/undercloud/home/zuul/undercloud_install.log

2021-07-06 18:51:55 | "tripleo_common.image.exception.ImageNotFoundException: Not found image: http://104.239.168.195:5001/v2/tripleotraincentos8/centos-binary-ironic-conductor/manifests/2a36198669a503670ae6f6e81ea95136"

there are many more lines in this log, but it really looks like the same issue.

[Cédric Jeanneret (cjeanner)]

Ah.. well.... ALL branches.

The same change is backported down to train, and we're seeing that same job failing everywhere:
https://review.opendev.org/q/I55905cb299ae9e35b605f0a9219177f809d6919e

[Marios Andreou (marios-b)]

suspected proxy issue is resolving - we had a similar bug last sprint there fyi https://bugs.launchpad.net/tripleo/+bug/1934097

[wes hayutin (weshayutin)]

let's get this on Alan's radar..