tripleo-ci-centos-8-scenario007-multinode-oooq-container is having intermittently timeouts

[Pooja Jadhav]

tripleo-ci-centos-8-scenario007-multinode-oooq-container is having intermittently timeouts with below trace back :

2020-06-16 21:05:06.637598 | primary | TASK [overcloud-deploy : echo deployment_status] *******************************
2020-06-16 21:05:06.637612 | primary | Tuesday 16 June 2020 21:05:06 +0000 (0:00:00.081) 0:36:09.701 **********
2020-06-16 21:05:06.707396 | primary | ok: [undercloud] => {
2020-06-16 21:05:06.707440 | primary | "overcloud_deploy_result": "passed"
2020-06-16 21:05:06.707453 | primary | }
2020-06-16 21:05:06.725636 | primary |
2020-06-16 21:05:06.725677 | primary | TASK [overcloud-deploy : Copy overcloudrc to ansible host] *********************
2020-06-16 21:05:06.725690 | primary | Tuesday 16 June 2020 21:05:06 +0000 (0:00:00.088) 0:36:09.789 **********
2020-06-16 21:05:07.497925 | primary | changed: [undercloud]
2020-06-16 21:05:07.513037 | primary |
2020-06-16 21:05:07.513081 | primary | TASK [overcloud-deploy : Run post-deploy script] *******************************
2020-06-16 21:05:07.513103 | primary | Tuesday 16 June 2020 21:05:07 +0000 (0:00:00.787) 0:36:10.577 **********
2020-06-16 21:05:17.055668 | primary | changed: [undercloud]
2020-06-16 21:05:17.059996 | primary |
2020-06-16 21:05:17.060209 | primary | PLAY [Check the result of the deployment] **************************************
2020-06-16 21:05:17.089701 | primary |
2020-06-16 21:05:17.089753 | primary | TASK [ensure the deployment result has been read into memory] ******************
2020-06-16 21:05:17.089768 | primary | Tuesday 16 June 2020 21:05:17 +0000 (0:00:09.575) 0:36:20.153 **********
2020-06-16 22:44:13.489333 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2020-06-16 22:44:13.493807 | POST-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/post.yaml@master]
2020-06-16 22:44:25.114275 |

Reference Links :

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario007-multinode-oooq-container

https://4c474afa65738b153370-596781338ef27878bb1588affd1dfa8a.ssl.cf2.rackcdn.com/735428/4/check/tripleo-ci-centos-8-scenario007-multinode-oooq-container/36084fe/job-output.txt

[yatin (yatinkarel)]

The same issue got hit into Downstream Testing, So bit of investigation done in https://review.opendev.org/#/c/736468/ and it's found that it do not happens when using single playbook multinode.yml instead of using all playbooks. This differs in having multiple ansible-playbook call vs single ansible-playbook call, The root cause is not clear why it happens with multiple playbooks, but using single playbook should clear the issue, will propose the patch by removing DNM.

[yatin (yatinkarel)]

Also seeing in other jobs, like https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates&branch=stable%2Ftrain, https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades etc, there is WIP patch up to clear these:- https://review.opendev.org/c/openstack/tripleo-ci/+/765297

[Sagi (Sergey) Shnaidman (sshnaidm)]

Fixed by https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770924

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sorin Sbârnea <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/770766
Reason: Abandoning as I suppose that was a leftover. Resurrect it if not.