periodic-...-1ctlr-featureset010-master periodic-...-multinode-1ctlr-featureset037-updates-master jobs failing "Not found image: ... tripleomaster/centos-binary-zaqar"

[Marios Andreou]

The periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master and periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master jobs are failing during undercloud install with an error "Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar" trace from [1] [2] looks like:

  2018-12-13 00:34:32 | Exception: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar:96234fdcee50d5326f9d942cc507e6ab76e78e2d_e92c7b37
  2018-12-13 00:34:32 | Traceback (most recent call last):
  2018-12-13 00:34:32 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1150, in _standalone_deploy
  2018-12-13 00:34:32 | parsed_args)
  2018-12-13 00:34:32 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 733, in _deploy_tripleo_heat_templates
  2018-12-13 00:34:32 | self._prepare_container_images(env, parsed_args)
  ...

  2018-12-13 00:34:32 | result = fn(*args, **kwargs)
  2018-12-13 00:34:32 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 399, in _inspect
  2018-12-13 00:34:32 | image_url.geturl())
  2018-12-13 00:34:32 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar:96234fdcee50d5326f9d942cc507e6ab76e78e2d_e92c7b37

[1] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/a963177/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-12-13_06_28_23
[2] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/f4da5ce/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-12-13_06_45_36

[Martin André (mandre)]

Possibly the image with the given tag was removed from trunk.registry.rdoproject.org/tripleomaster to make some space?

$ skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar:96234fdcee50d5326f9d942cc507e6ab76e78e2d_e92c7b37
FATA[0002] Error reading manifest 96234fdcee50d5326f9d942cc507e6ab76e78e2d_e92c7b37 in trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar: manifest unknown: manifest unknown

$ skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar:current-tripleo
{
    "Name": "trunk.registry.rdoproject.org/tripleomaster/centos-binary-zaqar",
    "Digest": "sha256:39660b6dca47901aab343730cc0e854811215b5126fabfc8da32d41b22a80b06",
    "RepoTags": [
        "6b7496a7d2aa5c1092ac4a7f72582b110d270324_0d30b916",
        "965941f1e62cef16967e7a7cd6d98263e52acb62_0989b280",
        "current-tripleo",
        "current-tripleo-rdo",
        "current-tripleo-rdo-internal",
        "d8bae7b99708a308fb52ec13ae71f5a5f42a0a5d_e972080c",
        "tripleo-ci-testing"
    ],
    "Created": "2018-12-04T05:43:07.346639643Z",
[snip]

[Marios Andreou (marios-b)]

bump. digging here as it is blocking promotions (also reached out to some folks on irc for now) but not making any progress. The issue continues today [1][2].

The only thing I came across (but not sure at all of relevance) is a failed build on the master dlrn report [3] - the failing 'zaqar-status' merged 12th [4] but again, this is dlrn packaging so not relevant to the missing container.

[1] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/e06cc50/logs/undercloud/home/zuul/install-undercloud.log.txt.gz
[2] http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/54de249/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[3] https://trunk.rdoproject.org/centos7-master-head/37/fc/37fc134d1102adaae2d7097c47669e1f8d52afb9_6f7d6f65/rpmbuild.log
[4] https://github.com/openstack/zaqar/commit/37fc134d1102adaae2d7097c47669e1f8d52afb9

[Marios Andreou (marios-b)]

15:14 < ykarel> marios_|ruck, weshay https://review.openstack.org/#/c/610929/ cause the https://bugs.launchpad.net/tripleo/+bug/1808349

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/625256

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/625259

[Marios Andreou (marios-b)]

15:10 < ykarel> marios_|ruck, what is blocking zaqar -> zaqar-wsgi image copy?
15:10 < ykarel> https://review.openstack.org/#/c/625256/3/container-images/overcloud_containers.yaml.j2@890

15:13 < ykarel> marios_|ruck, this is chicken-egg situation
15:13 < ykarel> marios_|ruck, so to get the patch merged we need a image docker.io/tripleomaster/centos-zaqar-wsgi:current-tripleo
15:13 < marios_|ruck> ykarel: so do we need to rename the currently running zaqar container image to be called zaqar-wsgi
15:14 < ykarel> if u rename other jobs will fail, so u need to create a copy of zaqar:current-tripleo to zaqar-wsgi:current-tripleo at
                docker.io
15:14 < ykarel> marios_|ruck, ^^

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.openstack.org/625259

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625256
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=3a44e5bed29b34c98beb8c06e8ecaef24b56ab57
Submitter: Zuul
Branch: master

commit 3a44e5bed29b34c98beb8c06e8ecaef24b56ab57
Author: Marios Andreou <email address hidden>
Date: Fri Dec 14 15:37:57 2018 +0200

    Replace zaqar image with zaqar-wsgi image

    Since kolla change Ic356fb81c19ebbc78aa3c7e871c4946d1f712a45 there are
    now 2 zaqar images, and we only need the WSGI one.

    Change-Id: I26e3aca0641d093857da5da5e88da004ff4e1576
    Co-Authored-By: Emilien Macchi <email address hidden>
    Closes-Bug: #1808349

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 10.3.0 release.