Test result sorting doesn't always sort failed test classes at the top of the HTML page

[melanie witt]

I noticed a case where the sorting doesn't put a test class with a failure at the top of the HTML result page.

Example:

nova.tests.unit.api.test_auth.TestNovaKeystoneContextMiddleware 6 5 1 0 0 Detail
test_user_id_trumps_user pass
test_invalid_service_catalog pass
test_user_only fail
test_request_id_extracted_from_env pass
test_no_user_or_user_id pass
test_user_id_only pass

Because the first test in the class passes, the class was considered to be a "passing class" and was not sorted to the top of the results even though it had a failure in the third test. The current logic only considers the first occurrence of a test class and doesn't update a test class's status after that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/396350

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396350
Committed: https://git.openstack.org/cgit/openstack/os-testr/commit/?id=b296846535ab7df1d1fd1136db757dd1196c96cb
Submitter: Jenkins
Branch: master

commit b296846535ab7df1d1fd1136db757dd1196c96cb
Author: melanie witt <email address hidden>
Date: Thu Nov 10 18:13:26 2016 +0000

    Track failures during sorting of test results

    The current logic only considers the first occurrence of a test class
    and doesn't update a class's pass/fail status after that. So a test
    class, for example:

     TestClass
       test_1 pass
       test_2 fail
       test_3 pass

    is counted as a passing test class even though it has a failure.

    This adds failure tracking to the test result sorting so that any
    failure in a test class will make it considered a failing test class
    and be sorted at the top of the HTML result page.

    Closes-Bug: #1640889

    Change-Id: I06919336a09c4afda8ec3a9e7d64d305fbd169c3