manila-tempest-dsvm-scenario job does not fail if individual tests fail

[Goutham Pacha Ravi]

A failure that was reported as a successful job run:

Job Results: https://review.openstack.org/#/c/544519/12
Failures: http://logs.openstack.org/19/544519/12/check/manila-tempest-dsvm-scenario/0c5b584/logs/testr_results.html.gz (gz of all logs attached below)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/568650

[Goutham Pacha Ravi (gouthamr)]

Another instance of this issue:

All tempest API test jobs are meant to fail on https://review.openstack.org/#/c/568393/1 because the code change that it tests hasn't merged in manila yet (https://review.openstack.org/#/c/568364/). However, some of the jobs posted "SUCCESS", despite test failures.

Example:
http://logs.openstack.org/93/568393/1/check/manila-tempest-dsvm-postgres-container/366dc5c/job-output.txt.gz

The RCA:
- The issue occurs because we don't seem to be saving the execution result of the tempest test invocation correctly. Please see https://review.openstack.org/568650 for the fix.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/568650
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=a0947ab42d9f37236c1f0784bd39346b04f8b6a8
Submitter: Zuul
Branch: master

commit a0947ab42d9f37236c1f0784bd39346b04f8b6a8
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue May 15 10:36:08 2018 -0700

    Fix post-execution for tempest tests

    Manila's post_test_hook runs tempest after
    populating tempest.conf with the requisite
    options, the results from tempest are saved
    prior to performing some post tasks, including
    deciding to run tests again with a different
    configuration if necessary
    (Ex: manila-tempest-minimal-dsvm-dummy). The
    result is then replayed at the end of the gate
    job so it can be reported to gerrit as a
    SUCCESS or a FAILURE.

    Fix the mis-aligned command that saves the
    test execution status to replay later.

    Change-Id: I1578760697aeae6e20bf567c60fbcdee0a642bee
    Closes-Bug: 1770183

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 7.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/578772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/578772
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=a6f4e23d26b7e7e054ad29ea572a0f23f83083a5
Submitter: Zuul
Branch: stable/queens

commit a6f4e23d26b7e7e054ad29ea572a0f23f83083a5
Author: Goutham Pacha Ravi <email address hidden>
Date: Tue May 15 10:36:08 2018 -0700

    Fix post-execution for tempest tests

    Manila's post_test_hook runs tempest after
    populating tempest.conf with the requisite
    options, the results from tempest are saved
    prior to performing some post tasks, including
    deciding to run tests again with a different
    configuration if necessary
    (Ex: manila-tempest-minimal-dsvm-dummy). The
    result is then replayed at the end of the gate
    job so it can be reported to gerrit as a
    SUCCESS or a FAILURE.

    Fix the mis-aligned command that saves the
    test execution status to replay later.

    Change-Id: I1578760697aeae6e20bf567c60fbcdee0a642bee
    Closes-Bug: 1770183
    (cherry picked from commit a0947ab42d9f37236c1f0784bd39346b04f8b6a8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.0.2 release.