tripleo-ansible molecule jobs get stuck with no output

[Sorin Sbarnea]

Apparently molecule zuul jobs from ansible-tripleo repository can get stuck forever and without giving any feedback, causing a chain of retries and no long left behind.

I was able to spot the place where they get stuck:

2021-03-31 11:12:42.617936 | TASK [Run role test job]
2021-03-31 11:12:44.145599 | centos-8 | Ansible test environment is now active
2021-03-31 11:12:44.145709 | centos-8 | Run 'unset-ansible-test-env' to deactivate.
2021-03-31 11:12:44.145734 | centos-8 |
2021-03-31 11:12:44.157905 | centos-8 | Running scope as unit: run-r4bf5dfd5321f43f8a0a116d54cbf0e2c.scope
2021-03-31 11:12:44.862134 | centos-8 | ============================= test session starts ==============================
2021-03-31 11:12:44.862214 | centos-8 | platform linux -- Python 3.6.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
2021-03-31 11:12:44.862231 | centos-8 | rootdir: /home/zuul/src/opendev.org/openstack/tripleo-ansible
2021-03-31 11:12:44.862249 | centos-8 | plugins: cov-2.11.1, metadata-1.11.0, html-3.1.1, forked-1.3.0, testinfra-6.2.0, testinfra-6.0.0, xdist-2.2.1
2021-03-31 11:12:44.862264 | centos-8 | collected 1 item
2021-03-31 11:12:44.862277 | centos-8 |
2021-03-31 11:29:49.412785 | [Zuul] Log Stream did not terminate
2021-03-31 11:29:49.413518 | centos-8 | ERROR
2021-03-31 11:29:49.413911 | centos-8 | {
2021-03-31 11:29:49.414003 | centos-8 | "msg": "Data could not be sent to remote host \"104.130.239.177\". Make sure this host can be reached over ssh: ssh: connect to host 104.130.239.177 port 22: Connection timed out\r\n",
2021-03-31 11:29:49.414120 | centos-8 | "unreachable": true
2021-03-31 11:29:49.414242 | centos-8 | }
2021-03-31 11:29:49.461966 |
2021-03-31 11:29:49.462261 | PLAY RECAP
2021-03-31 11:29:49.462450 | centos-8 | ok: 2 changed: 0 unreachable: 1 failed: 0 skipped: 1 rescued: 0 ignored: 0
2021-03-31 11:29:49.462590 |
2021-03-31 11:29:49.950477 | RUN END RESULT_UNREACHABLE: [untrusted : opendev.org/openstack/tripleo-ansible/zuul.d/playbooks/run.yml@master]
2021-03-31 11:29:49.950756 | Running Ansible cleanup...
2021-03-31 11:29:49.951123 | CLEANUP-RUN START: [trusted : opendev.org/opendev/base-jobs/playbooks/base/cleanup.yaml@master]
2021-03-31 11:29:52.561904 |
2021-03-31 11:29:52.562151 | PLAY [all]
2021-03-31 11:29:52.697477 |
2021-03-31 11:29:52.697774 | TASK [get df disk usage]

It seems that something is happening during this task execution that makes the host unreachable to zuul.