tempest is no longer failing appropriately in oooq

[wes hayutin]

The following test job should be failing on tempest but it is not [1 -2]

I've traced the issue to ostestr [3]
When the options --no-pretty --subunit are used the script will *always* return 0
When the options --pretty are used the script will fail appropriately

I also noticed the parentheses allow the ostestr command to work appropriately with --no-pretty and --subunit and fail when tempest hits a failure.

The issue may have been introduced here [4], because when the command runs in with code prior to that commit tests fail appropriately [5]

This is a critical bug and needs immediate attention
Thanks!

[1] http://logs.openstack.org/02/491102/1/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/affbe80/logs/undercloud/home/jenkins/tempest/tempest.html.gz
[2] http://logs.openstack.org/02/491102/1/check/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/affbe80/console.html#_2017-08-05_00_17_11_168385
[3] https://docs.openstack.org/os-testr/latest/user/ostestr.html
[4] https://github.com/openstack/tripleo-quickstart-extras/commit/6d316c6bcc9b9622de88d90abb02bda2ed41c852
[5] http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq-master/8d8a9bf/logs/undercloud/home/jenkins/tempest-setup.sh.txt.gz
[5] http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha-tempest-oooq-master/8d8a9bf/console.html#_2017-08-04_09_28_07_716038

[Emilien Macchi (emilienm)]

Should be fixed by https://review.openstack.org/#/c/491189

[Arx Cruz (arxcruz)]

https://review.openstack.org/#/c/491170/ fix the problem

[Emilien Macchi (emilienm)]

Patch isn't merged yet - we'll set Fix Released once it's merged.

[Ben Nemec (bnemec)]

I'm dropping the alert from this. While I agree it's a serious problem, it isn't actively blocking anything so we don't need to notify people of it.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/491170
Reason: Need to clear out the queue as this is going to fail. Will restore momentarily

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/491170
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=b0b7f95435675446a9639eb4f69b596e4e628555
Submitter: Jenkins
Branch: master

commit b0b7f95435675446a9639eb4f69b596e4e628555
Author: Chandan Kumar <email address hidden>
Date: Sat Aug 5 14:08:43 2017 +0530

    Remove --subunit --no-pretty from tempest run

    * in tempest-results task we are already generating subunit.
      and --subunit --no-pretty also generates the subunit results.
      So, It i better to generate subunit at one place and use it
      from there.

    Closes-Bug: #1708793
    Change-Id: I5a7754932ea51a4eb34e92f05f9a212e587d8492

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.