openstack-periodic-latest-released pipeline tripleo-ci-centos-7* jobs failed with ImageNotFoundException

[Sandeep Yadav]

Description:-
===========

openstack-periodic-latest-released pipeline tripleo-ci-centos-7* jobs failed with ImageNotFoundException

Error Snippet:-
============

https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario007-standalone-train/ebc3656/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 967, in filter_images_with_labels
2020-03-31 10:25:18 | url, session=session)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 563, in _image_labels
2020-03-31 10:25:18 | i = cls._inspect(image_url, session)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 2332, in _inspect
2020-03-31 10:25:18 | image_url, session=session)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f
2020-03-31 10:25:18 | return self.call(f, *args, **kw)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call
2020-03-31 10:25:18 | do = self.iter(retry_state=retry_state)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter
2020-03-31 10:25:18 | return fut.result()
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
2020-03-31 10:25:18 | return self.__get_result()
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call
2020-03-31 10:25:18 | result = fn(*args, **kwargs)
2020-03-31 10:25:18 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 723, in _inspect
2020-03-31 10:25:18 | image_url.geturl())
2020-03-31 10:25:18 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleotrain/centos-binary-cron:ac250e5ab0b73ae2ecfb5eb6b199602a0f2f52d3_50a5446c
~~~

https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-train/b72f9b5/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
2020-03-31 10:24:59 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleotrain/centos-binary-cron:ac250e5ab0b73ae2ecfb5eb6b199602a0f2f52d3_50a5446c
~~~

https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-train/b1e091e/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
2020-03-31 10:25:46 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleotrain/centos-binary-aodh-api:ac250e5ab0b73ae2ecfb5eb6b199602a0f2f52d3_50a5446c
~~~

Investigation:-
==============

* It looks like container image build job pushed containers with different hash and deploy jobs is using different hash

https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/0e23089/job-output.txt

~~~
2020-03-31 09:17:04.905257 | TASK [build-containers : Get contents of openstack repo baseurl for the version hash]
2020-03-31 09:17:05.219686 | primary | http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos7-train/86/9f/869f68fff649260b19657c20909d3843c50c9f83_c0bb2d14
~~~

In the above example: container image build job pushed containers - with _c0bb2d14 hash , but deploy jobs using 50a5446c

[yatin (yatinkarel)]

The issue happened as at the time container build job ran mirror was not refreshed so it build containers with old hash and then when deploy job ran mirrors were updated and different hash was picked in tripleo-ci-testing. If jobs are run right now issue shouldn't been seen as at current stage mirrors are in sync[1]. https://review.rdoproject.org/r/#/c/26180/ should avoid the issue in future as in container build job mirror will not be used after the revert.

[1] http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos7-train/tripleo-ci-testing/delorean.repo vs http://trunk.rdoproject.org/centos7-train/tripleo-ci-testing/delorean.repo

[yatin (yatinkarel)]

https://review.rdoproject.org/r/#/c/26180/ merged.