matching centos and fedora hashes, however tripleo-ci-testing pin fails

[wes hayutin]

We have matching centos and fedora hashes however we get this error:

2019-01-08 22:47:25.483706 | TASK [Promote consistent to tripleo-ci-testing for fedora repo if an equivalent exists]
2019-01-08 22:47:25.086783 | rdo-centos-7 | Looking in indexes: http://mirror.regionone.rdo-cloud.rdoproject.org/pypi/simple, http://mirror.regionone.rdo-cloud.rdoproject.org/wheel/centos-7.6.1810-x86_64
2019-01-08 22:47:25.265637 | rdo-centos-7 | Requirement already up-to-date: dlrnapi_client in ./dlrnapi_venv/lib/python2.7/site-packages (0.5.1)
2019-01-08 22:47:25.302079 | rdo-centos-7 | Requirement already up-to-date: shyaml in ./dlrnapi_venv/lib/python2.7/site-packages (0.6.1)
2019-01-08 22:47:25.309092 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: six>=1.10 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (1.12.0)
2019-01-08 22:47:25.311416 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: urllib3>=1.15.1 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (1.24.1)
2019-01-08 22:47:25.325737 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: setuptools>=21.0.0 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (40.6.3)
2019-01-08 22:47:25.331780 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: certifi>=14.05.14 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (2018.11.29)
2019-01-08 22:47:25.334027 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.3 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (2.7.5)
2019-01-08 22:47:25.337730 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: pyyaml in ./dlrnapi_venv/lib/python2.7/site-packages (from shyaml) (3.13)
2019-01-08 22:47:29.998642 | rdo-centos-7 | d71c8b4aaafda38678dde99ddc5c17ed292b1fb5 and ae1c2bd0f102801e7ae33240cc15a44ecf9a911f hashes on centos does not have an equivalent in fedora. Skipping this run
2019-01-08 22:47:31.013064 | rdo-centos-7 | ERROR
2019-01-08 22:47:31.013305 | rdo-centos-7 | {
2019-01-08 22:47:31.013467 | rdo-centos-7 | "delta": "0:00:05.250789",
2019-01-08 22:47:31.013591 | rdo-centos-7 | "end": "2019-01-08 22:47:29.999180",
2019-01-08 22:47:31.013668 | rdo-centos-7 | "msg": "non-zero return code",
2019-01-08 22:47:31.013739 | rdo-centos-7 | "rc": 1,
2019-01-08 22:47:31.013795 | rdo-centos-7 | "start": "2019-01-08 22:47:24.748391"
2019-01-08 22:47:31.013848 | rdo-centos-7 | }
2019-01-08 22:47:31.026889 |
2019-01-08 22:47:31.026998 | PLAY RECAP

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/0cc95f1/job-output.txt.gz#_2019-01-08_22_47_31_013064

[yatin (yatinkarel)]

Ok so this difference mentioned in screenshots(lag between building same package) is expected with the approach used(matching 'consistent' hashes for Fedora and CentOS) because Fedora and CentOS dlrn builder are independent so can differ in timing in building packages.

I think we can skip(or atleast not FAIL the promotion-consistent-to-tripleo-ci-testing job) the check(comparing hashes), run both CentOS/Fedora promotion jobs, promote CentOS as usual, promote Fedora only if hash matches. In this approach Fedora might lag behind CentOS hash.

[Gabriele Cerami (gcerami)]

This happened again in the next run, it means that fedora hash is going to always be built a bit after centos. The hash is there, but it's there too late.
I'll have to add a polling cycle in the task, so we are not trying only once, and we fail after some attempts.

[Gabriele Cerami (gcerami)]

proposed https://review.rdoproject.org/api/18149