Define stestr facts conditaion check fails in validate-tempest role

[Martin Kopec]

The problem affects FS20 rocky promotion job due to a conditional check failure [1] in [2]:

 fatal: [undercloud]: FAILED! => {"msg": "The conditional check 'testr_dir.stat.exists' failed. The error was: error while evaluating conditional (testr_dir.stat.exists): 'ansible.parsing.yaml.objects.AnsibleUnicode object' has no attribute 'stat'\n\nThe error appears to have been in '/home/zuul/workspace/.quickstart/usr/local/share/ansible/roles/validate-tempest/tasks/stackviz.yml': line 13, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Define stestr facts\n ^ here\n"}

The failed init of stestr makes Generate testrepository.subunit results file task failed due to testr command not found [3] which probably makes Generating HTML result file failed as well [4]. Another occurrence [5].

The issue could have been seen in FS 16 master couple days ago [6], however it seems to be working now.

[1] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/6760f43/job-output.txt.gz#_2018-10-01_21_03_01_762184

[2] https://github.com/openstack/tripleo-quickstart-extras/blob/59e30de7a689a125a34bae0a195c3bc1d70b2202/roles/validate-tempest/tasks/stackviz.yml#L13

[3] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/375cc2f/job-output.txt.gz#_2018-10-02_16_04_08_857491

[4] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/375cc2f/job-output.txt.gz#_2018-10-02_16_04_09_907556

[5] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/375cc2f/job-output.txt.gz#_2018-10-02_16_03_57_709945

[6] https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/2723047/job-output.txt.gz#_2018-09-20_02_35_18_932093

[Marios Andreou (marios-b)]

master gate check hitting this today too http://logs.openstack.org/98/604298/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/037bd06/job-output.txt.gz#_2018-10-04_04_39_49_234612 2018-10-04 04:39:48.087051

primary | TASK [validate-tempest : Exit with tempest result code if configured] **********
2018-10-04 04:39:48.127135 | primary | Thursday 04 October 2018 04:39:48 +0000 (0:00:02.239) 0:04:26.387 ******
2018-10-04 04:39:49.234612 | primary | fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "tail -10 tempest.log; exit 1", "delta": "0:00:00.012430", "end": "2018-10-04 04:39:49.184002", "msg": "non-zero return code", "rc": 1, "start": "2018-10-04 04:39:49.171572", "stderr": "", "stderr_lines": [], "stdout": "2018-10-04 04:39:18 | - Skipped: 0\n2018-10-04 04:39:18 | - Expected Fail: 0\n2018-10-04 04:39:18 | - Unexpected Success: 0\n2018-10-04 04:39:18 | - Failed: 1\n2018-10-04 04:39:18 | Sum of execute time for each test: 23.6150 sec.\n2018-10-04 04:39:18 | \n2018-10-04 04:39:18 | ==============\n2018-10-04 04:39:18 | Worker Balance\n2018-10-04 04:39:18 | ==============\n2018-10-04 04:39:18 | - Worker 0 (1 tests) => 0:00:23.614971", "stdout_lines": ["2018-10-04 04:39:18 | - Skipped: 0", "2018-10-04 04:39:18 | - Expected Fail: 0", "2018-10-04 04:39:18 | - Unexpected Success: 0", "2018-10-04 04:39:18 | - Failed: 1", "2018-10-04 04:39:18 | Sum of execute time for each test: 23.6150 sec.", "2018-10-04 04:39:18 | ", "2018-10-04 04:39:18 | ==============", "2018-10-04 04:39:18 | Worker Balance", "2018-10-04 04:39:18 | ==============", "2018-10-04 04:39:18 | - Worker 0 (1 tests) => 0:00:23.614971"]}

[Marios Andreou (marios-b)]

i think this is a duplicate for 1793665 btw since the error being referred to here has ignore_errors on it (you can see from the trace in [1] of the description above) some more context from irc just now:

11:38 < marios|rover> kopecmartin|ruck: no i think that testr you point to is invalid since it has ignore on it?
11:39 < marios|rover> kopecmartin|ruck: here
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/6760f43/job-output.txt.gz#_2018-10-01_21_03_01_762184
11:39 < marios|rover> kopecmartin|ruck: yes it fails but then you see it has 2018-10-01 21:03:01.814479 | tripleo-ovb-centos-7 | ...ignoring
11:39 < marios|rover> kopecmartin|ruck: and if you scroll down
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/6760f43/job-output.txt.gz#_2018-10-01_21_03_22_326496
11:40 < marios|rover> you see the same error which is the actual issue

[Martin Kopec (mkopec)]

Fixed here: https://review.openstack.org/#/c/605419/