standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source

[Marios Andreou]

The undercloud-upgrade-victoria at [1][2] and the standalone-upgrade-victoria at [3][4] are failing during the upgrade with traces like:

(undercloud-upgrade-victoria)
        * tripleo_common.image.exception.ImageNotFoundException: Not found image: http://23.253.57.32:5001/v2/tripleovictoria/openstack-keystone/manifests/7f974e10d7184d5fc45445a3073333bd"], "stdout": "", "stdout_lines": []}

(standalone-upgrade-victoria [4] )
        * 2020-11-16 22:36:01 | 2020-11-16 22:36:01.357069 | bc764e10-3527-7de4-b0c9-000000000783 | FATAL | Pull latest cinder_volume images | standalone | error={"attempts": 3, "changed": true, "cmd": ["podman", "pull", "23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd"], "delta": "0:00:00.128602", "end": "2020-11-16 22:36:01.321355", "msg": "non-zero return code", "rc": 125, "start": "2020-11-16 22:36:01.192753", "stderr": "Trying to pull 23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd...\n Get https://23.253.57.57:5001/v2/: http: server gave HTTP response to HTTPS client\nError: error pulling image \"23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd\": unable to pull 23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd: unable to pull image: Error initializing source docker://23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd: error pinging docker registry 23.253.57.57:5001: Get https://23.253.57.57:5001/v2/: http: server gave HTTP response to HTTPS client", "stderr_lines": ["Trying to pull 23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd...", " Get https://23.253.57.57:5001/v2/: http: server gave HTTP response to HTTPS client", "Error: error pulling image \"23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd\": unable to pull 23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd: unable to pull image: Error initializing source docker://23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd: error pinging docker registry 23.253.57.57:5001: Get https://23.253.57.57:5001/v2/: http: server gave HTTP response to HTTPS client"], "stdout": "", "stdout_lines": []}

Looking at the inventory it looks like the upgrade is using the ussuri registry instead of the victoria one so we have a nit somewhere that references 'registry_ip_address_branch'. For example [5] which matches the standalone upgrade trace above:

    * provider_job_branch: ussuri
      registry_ip_address: 23.253.57.57
      registry_ip_address_branch:
        ussuri: 23.253.57.57
        victoria: 23.253.57.47

The job is failing to pull docker://23.253.57.57:5001/tripleovictoria/openstack-cinder-volume:7f974e10d7184d5fc45445a3073333bd which is the *ussuri* registry

These are new jobs but since they block victoria check/gate we can mark them non voting until we investigate and close this bug.

[1] https://7085ed736f9157d6d701-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf1.rackcdn.com/762658/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/ad84735/logs/undercloud/home/zuul/undercloud_upgrade.log
[2] https://7085ed736f9157d6d701-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf1.rackcdn.com/762658/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/ad84735/logs/undercloud/var/log/tripleo-container-image-prepare.log
[3] https://f6e3218ace9cacf32740-5fc15b0c04646205dc30781d5a18c6a5.ssl.cf5.rackcdn.com/762890/1/check/tripleo-ci-centos-8-standalone-upgrade-victoria/4071044/logs/undercloud/home/zuul/standalone_upgrade.log
[4] https://a1af8c3f8f645c717fba-41571b934cfe7bb61559f346800eddd4.ssl.cf1.rackcdn.com/762900/3/check/tripleo-ci-centos-8-standalone-upgrade-victoria/b902a76/logs/undercloud/home/zuul/standalone_upgrade.log
[5] https://a1af8c3f8f645c717fba-41571b934cfe7bb61559f346800eddd4.ssl.cf1.rackcdn.com/762900/3/check/tripleo-ci-centos-8-standalone-upgrade-victoria/b902a76/zuul-info/inventory.yaml

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/763005

[Marios Andreou (marios-b)]

testing https://review.opendev.org/763005 with https://review.rdoproject.org/r/31112

[Marios Andreou (marios-b)]

at the test https://review.rdoproject.org/r/#/c/31112/

tripleo-ci-centos-8-standalone-upgrade-victoria SUCCESS in 1h 58m 55s
tripleo-ci-centos-8-undercloud-upgrade-victoria SUCCESS in 1h 21m 19s

so now waiting for https://review.opendev.org/763005 to slay the zuul

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.opendev.org/763005
Reason: going to abandon and restore to reset the current check run failed on content provider

WTF

        * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6bd/763005/1/check/tripleo-ci-centos-8-content-provider/6bd7816/job-output.txt
        * 2020-11-19 08:04:05.544465 | primary | curl: (22) The requested URL returned error: 503 Service Unavailable
        * 2020-11-19 08:04:05.074284 | primary | TASK [repo-setup : Get DLRN hash newest - passed tag - component-based] ********

[Marios Andreou (marios-b)]

https://review.opendev.org/763005 still fighting to merge

fails so far in gate on unrelated and transient issues

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/763005
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/abd2442f228ac58b079ead30c11a5faef4f74eb8
Submitter: Zuul
Branch: master

commit abd2442f228ac58b079ead30c11a5faef4f74eb8
Author: MSA <email address hidden>
Date: Tue Nov 17 14:54:42 2020 +0200

    Fix nit in victoria release file using the wrong docker registry

    We must always reference the registry_ip_address_branch otherwise
    we can end up using the wrong registry as was the case for related
    bug.

    Change-Id: I911ff5dd7128b77727d16ec04463d98713046823
    Related-Bug: 1904565

[Marios Andreou (marios-b)]

 the fix merged at https://review.opendev.org/763005

jobs are now back to green - top 4 are green in

        * https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-victoria
        * https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade-victoria

moving fix-released

[Marios Andreou (marios-b)]

posted related tripleo-ci master: Remove the registry_ip_address variable from content providers https://review.opendev.org/c/openstack/tripleo-ci/+/764359

[Marios Andreou (marios-b)]

fix from comment #8 also related to another bug https://bugs.launchpad.net/tripleo/+bug/1905942/comments/8