periodic-... containers-build-push job skips retagging with tripleo-ci-testing

[Marios Andreou]

As discussed at https://bugs.launchpad.net/tripleo/+bug/1818994 comments 4/5/6 the periodic-tripleo-centos-7-master-containers-build-push job isn't retagging the containers correctly.

Inspection via skopeo confirms something is wrong comparing for example:

    skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-rabbitmq:1ac63709436a0230f547040e4a514470a3c19d78_9c2c4c8f
    ==> "Created": "2019-03-11T15:00:25.259008772Z",

    skopeo inspect docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-rabbitmq:tripleo-ci-testing
    ==> "Created": "2019-03-07T22:38:02.787234492Z",

I suspect it must be the conditional at https://review.openstack.org/#/c/641348/13/playbooks/tripleo-buildcontainers/run.yaml@89

=============================
[EDIT - 18Mar19]: to be clear we did not end up re-tagging with tripleo-ci-testing. Instead we changed the https://review.openstack.org/#/c/643187/ overcloud-pre-containers and deploy roles to use the container image tag correctly.

[Marios Andreou (marios-b)]

09:03 < marios> panda: ykarel damn it its a typo yaml vs yml
09:03 * marios facepalm
09:03 < marios> fixing
09:03 < ykarel> omg :)
09:03 < marios> https://git.openstack.org/cgit/openstack-infra/tripleo-ci/commit/?id=0f61e33f01886e3fbf36e7af4110e11a9e4f80bb&context=3&ignorews=0&dt=0
09:04 < marios> ykarel: :/
09:04 < marios> o_O
09:04 < marios> ykarel: yeah see in the diff there we add tag.yaml but include tag.yml
09:04 < marios> don't know why it didn't fail though on the include
09:05 < ykarel> marios, may be built_images returned blank list
09:05 < marios> ykarel: hmm :/ that is also not good
09:05 < marios> ykarel: if that is the case though it says 'changed' for that one
09:05 < marios> (i mean in console and also in ara )
09:06 < marios> whereas the tag is skipped
09:06 < ykarel> changed can be for blank [] as well
09:06 < marios> ykarel: k well we'll find out
09:06 < ykarel> ack
09:06 < marios> i think next periodic run in 2 hours maybe won't make that one though
09:06 < ykarel> hmm

09:07 < openstackgerrit> Marios Andreou proposed openstack-infra/tripleo-ci master: Fix typo in the retagging for periodic containers build job https://review.openstack.org/642662

remote: https://review.rdoproject.org/r/19131 Run containers push job test retag with version_hash

09:17 < openstackgerrit> Marios Andreou proposed openstack-infra/tripleo-ci master: DO NOT MERGE testing https://review.openstack.org/642662 https://review.openstack.org/642663

[Marios Andreou (marios-b)]

this was discussed during the ci community call just now. question raised by weshay is basically do we care that containers are not being tagged with 'tripleo-ci-testing'

Since https://review.openstack.org/641348 the containers pushed to rdo registry with the periodic containers build job are tagged correctly with the tripleo-ci-testing delorean repo hash.

So the question is, do we care that containers are not being pushed with 'tripleo-ci-testing' tag. If all the periodics/singlenode/multinode/ovb are using the tripleo-ci-testing repo delorean hash (and not 'tripleo-ci-testing' itself as a tag).

Removing the promotion blocker and will rework https://review.openstack.org/642662 to instead remove the tagging altogether, unless someone tells us we really need to tag with 'tripleo-ci-testing'

[wes hayutin (weshayutin)]

The human readable tag in periodic jobs is not a requirement

[Marios Andreou (marios-b)]

well, we might still care. looks like ovb is still pulling with "tripleo-ci-testing"

2019-03-13 01:05:31 | TASK [tripleo-container-tag : Pull 192.168.24.1:8787/tripleomaster/centos-binary-cinder-volume:tripleo-ci-testing-updated-20190312234121 image] ***

http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/f5a872d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

---------------

other periodics look ok though e.g.

" periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master"

" baseurl=http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/centos7/9f/a2/9fa26e439b9064a64aa96d29594e26f71843db9f_3f92697b " ==> http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/ba37efe/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz

+ "tag: "9fa26e439b9064a64aa96d29594e26f71843db9f_3f92697b"" ==> http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/ba37efe/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz

+ "GET /v2/tripleomaster/centos-binary-mariadb/manifests /9fa26e439b9064a64aa96d29594e26f71843db9f_3f92697b HTTP/1.1" 200 6574" ==> http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master/ba37efe/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz

" periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master"

"baseurl=http://mirror.regionone.rdo-cloud.rdoproject.org:8080/rdo/centos7-rocky/f3/4e/f34e27928e17de3161919cfe908d73721d66a0b1_9ef0be25" @@ http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master/50a3d26/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz

"2019-03-13 00:43:04 | "2019-03-13 00:39:33,180 DEBUG: 24258 -- config_image docker.io/tripleorocky/centos-binary-barbican-api:f34e27928e17de3161919cfe908d73721d66a0b1_9ef0be25"," ==> http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master/50a3d26/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

" "rdo_version": "f34e27928e17de3161919cfe908d73721d66a0b1_9ef0be25"" ==> http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-master/50a3d26/logs/undercloud/var/log/extra/docker/containers/barbican_api/docker_info.log.txt.gz

[Marios Andreou (marios-b)]

discussin in irc just now with panda and weshay in freenode oooq

14:08 < marios> panda|rover: retag should be very simple in theory (like
                https://review.openstack.org/#/c/642662/5/playbooks/tripleo-buildcontainers/run.yaml) but we ran into ansible version
                issues so we need to bump something
14:09 < marios> weshay: well we need to decide what is faster but also safe enough to do quickly. fixup promotion script and
http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml
14:09 < marios> vs https://review.openstack.org/#/c/642662/5/playbooks/tripleo-buildcontainers/run.yaml

re-adding the blocker flag and bumping the priority it blocks promotions

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/643095

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: master
Review: https://review.openstack.org/643095
Reason: https://review.openstack.org/#/c/643187/ we went with that one

[Marios Andreou (marios-b)]

We are trying that instead https://review.openstack.org/#/c/643187/ and it merged let's see

(looks like was merged in a rush w/out the bug info so adding manually)