In most of the fs016 periodic jobs once tempest run finishes, the validate-tempest role fails to generate tempest.html result. It makes very hard to user to debug and go to check tempest.log file for the tempest errors.
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/2723047/job-output.txt.gz#_2018-09-20_02_35_29_598462
2018-09-20 02:35:29.598462 | primary | TASK [validate-tempest : Generate testrepository.subunit results file] *********
2018-09-20 02:35:29.615521 | primary | Thursday 20 September 2018 02:35:29 +0000 (0:00:00.672) 0:18:53.370 ****
2018-09-20 02:35:30.688127 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail && testr last --subunit | subunit-1to2 > /home/zuul/tempest/testrepository.subunit", "delta": "0:00:00.381209", "end": "2018-09-20 02:35:30.669697", "msg": "non-zero return code", "rc": 127, "start": "2018-09-20 02:35:30.288488", "stderr": "/bin/sh: testr: command not found", "stderr_lines": ["/bin/sh: testr: command not found"], "stdout": "", "stdout_lines": []}
2018-09-20 02:35:30.688196 | primary | ...ignoring
2018-09-20 02:35:30.712744 | primary |
2018-09-20 02:35:30.712900 | primary | TASK [validate-tempest : Generate HTML results file] ***************************
2018-09-20 02:35:30.733215 | primary | Thursday 20 September 2018 02:35:30 +0000 (0:00:01.117) 0:18:54.488 ****
2018-09-20 02:35:31.726561 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "set -o pipefail &&\n subunit2html $(find /home/zuul/tempest/.testrepository -name [0-9] | head -1) /home/zuul/tempest/tempest.html 2>&1 >> tempest.log", "delta": "0:00:00.360371", "end": "2018-09-20 02:35:31.706626", "msg": "non-zero return code", "rc": 1, "start": "2018-09-20 02:35:31.346255", "stderr": "find: ‘/home/zuul/tempest/.testrepository’: No such file or directory", "stderr_lines": ["find: ‘/home/zuul/tempest/.testrepository’: No such file or directory"], "stdout": "Traceback (most recent call last):\n File \"/usr/bin/subunit2html\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python2.7/site-packages/os_testr/subunit2html.py\", line 740, in main\n stream = open(subunit_file, 'rb')\nIOError: [Errno 2] No such file or directory: '/home/zuul/tempest/tempest.html'", "stdout_lines": ["Traceback (most recent call last):", " File \"/usr/bin/subunit2html\", line 10, in <module>", " sys.exit(main())", " File \"/usr/lib/python2.7/site-packages/os_testr/subunit2html.py\", line 740, in main", " stream = open(subunit_file, 'rb')", "IOError: [Errno 2] No such file or directory: '/home/zuul/tempest/tempest.html'"]}
I think we need to seperate the html generation part. otherwise it is very hard to go through logs and find issue.
FS 16/17 master seem to work now, however the issue can be found in FS20 rocky promotion job.
In 50% cases of legacy-
[1] https:/
[2] https:/
[3] https:/