master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline

[Marios Andreou]

periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing is timing out starting ~ yesterday at [1] last three runs from the openstack-periodic-master pipeline are timing out like:

            2019-12-17 16:11:56.711573 | TASK [promote-hash : Promote consistent to tripleo-ci-testing for secondary (RHEL) repo if an equivalent exists]
            2019-12-17 16:43:48.617670 | RUN END RESULT_TIMED_OUT: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/run.yaml@master]

see [2][3][4]. Same is also happening for train see [5][6]

[1] https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master
[2] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/cd9d701/job-output.txt
[3] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/af8b2a0/job-output.txt
[4] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/20dc2a1/job-output.txt
[5] http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/591181d/job-output.txt
[6] http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/9d6c400/job-output.txt

[Marios Andreou (marios-b)]

posted that for testing https://review.rdoproject.org/r/#/c/24241/ but

11:43 < marios|ruck> chkumar|rover: https://review.rdoproject.org/zuul/stream/65444c6f81a04d719b8452b04b77d66c?logfile=console.log
11:43 < marios|ruck> chkumar|rover: gonna time out :/
11:44 < marios|ruck> booooo
11:44 < chkumar|rover> yes
11:44 < chkumar|rover> I donot know what is going on there

[chandan kumar (chkumar246)]

After discussing Jpena, we found that the rdo current is not consistent yet, once the rdo repo gets consistent then it will work.

We need to add a check in our tooling, to make sure rdo current needs to be consistent.

[chandan kumar (chkumar246)]

May be this needs to be improved to https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/promote-hash/tasks/promote-secondary-distro.yaml#L6 to check for current and consistent comparison.

[chandan kumar (chkumar246)]

Few more observations:
http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-promote-consistent-to-tripleo-ci-testing/4a91560/job-output.txt

In train the job also got timeout, but it shows the message:
2019-12-17 09:12:53.097805 | TASK [promote-hash : Promote consistent to tripleo-ci-testing for secondary (RHEL) repo if an equivalent exists]
2019-12-17 09:12:53.556906 | rdo-centos-7 | DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
2019-12-17 09:12:53.667363 | rdo-centos-7 | Looking in indexes: http://mirror.regionone.rdo-cloud.rdoproject.org/pypi/simple, http://mirror.regionone.rdo-cloud.rdoproject.org/wheel/centos-7.7-x86_64
2019-12-17 09:12:53.827703 | rdo-centos-7 | Requirement already up-to-date: dlrnapi_client in ./dlrnapi_venv/lib/python2.7/site-packages (0.6.1)
2019-12-17 09:12:53.887671 | rdo-centos-7 | Requirement already up-to-date: shyaml in ./dlrnapi_venv/lib/python2.7/site-packages (0.6.1)
2019-12-17 09:12:53.896952 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: six>=1.10 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (1.13.0)
2019-12-17 09:12:53.901833 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: urllib3>=1.15.1 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (1.25.7)
2019-12-17 09:12:53.924077 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: setuptools>=21.0.0 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (42.0.2)
2019-12-17 09:12:53.931369 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: certifi>=14.05.14 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (2019.11.28)
2019-12-17 09:12:53.933914 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.3 in ./dlrnapi_venv/lib/python2.7/site-packages (from dlrnapi_client) (2.8.1)
2019-12-17 09:12:53.938257 | rdo-centos-7 | Requirement already satisfied, skipping upgrade: pyyaml in ./dlrnapi_venv/lib/python2.7/site-packages (from shyaml) (5.2)
2019-12-17 09:12:54.150365 | rdo-centos-7 | Checking existence of commit 6a616e5db26a18e9e9d842e3979de2ad3e394219 / distro 7b91151405fb7a6c0d2f1d403c625eaaba60fb72 in RHEL
2019-12-17 09:12:57.925391 | rdo-centos-7 | commit 6a616e5db26a18e9e9d842e3979de2ad3e394219 / distro 7b91151405fb7a6c0d2f1d403c625eaaba60fb72 combination not found in RHEL
2019-12-17 09:44:41.904603 | RUN END RESULT_TIMED_OUT: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/run.yaml@master]
2019-12-17 09:44:41.906324 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/post.yaml@master]
2019-12-17 09:44:46.848067 |

And in master it is behaving differently.

Other thing, we need to incre...

[Marios Andreou (marios-b)]

13:41 < marios|ruck> jpena: is there something known blocking the consistent build wrt https://bugs.launchpad.net/tripleo/+bug/1856830
13:41 <@openstack> Launchpad bug 1856830 in tripleo "master + train promote-consistent-to-tripleo-ci-testing times out blocks master+train pipeline" [Critical,Triaged]
13:41 < marios|ruck> jpena: we've seen that timeout for more than a day now
13:42 < jpena> marios|ruck: yes, there was a build in rhel8-master that failed (oslo-rootwrap) due to a unit test timeout. It was fixed earlier today

[chandan kumar (chkumar246)]

I think we have more issue there:
from recent run: http://logs.rdoproject.org/08/24208/3/check/periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/d625814/job-output.txt

2019-12-18 12:09:42.066012 | TASK [promote-hash : Promote consistent to tripleo-ci-testing for secondary (RHEL) repo if an equivalent exists]
2019-12-18 12:50:22.465753 | rdo-centos-7 | ERROR
2019-12-18 12:50:22.466182 | rdo-centos-7 | {
2019-12-18 12:50:22.466288 | rdo-centos-7 | "attempts": 40,
2019-12-18 12:50:22.466350 | rdo-centos-7 | "msg": "Unsupported parameters for (command) module: environment Supported parameters include: _raw_params, _uses_shell, argv, chdir, creates, executable, removes, stdin, warn, zuul_log_id"
2019-12-18 12:50:22.466420 | rdo-centos-7 | }

[Ronelle Landy (rlandy)]

Fixed by https://review.rdoproject.org/r/#/c/24246/