xfail doesn't take effect on a whole testcase failure

Bug #1568133 reported by Arata Notsu on 2016-04-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rally
Medium
Yaroslav Lobankov

Bug Description

If Tempest fails in a test's setUpClass(), the result that will be reported by Rally is "fail", even though the test is listed in a file passed to --xfails-file.

In my case, I deployed Keystone with the "templated" catalog driver. This driver, unlike the "sql" driver, doesn't support APIs for services or endpoints. So the test "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON.test_update_endpoint[id-37e8f15e-ee7c-4657-a1e7-f6b61e375eff,smoke]" is expected to fail. Actually it does, but the failure occurs in setUpClass() as you can see in the traceback:

Traceback (most recent call last):
  File tempest/test.py, line 280, in setUpClass
    six.reraise(etype, value, trace)
  File tempest/test.py, line 273, in setUpClass
    cls.resource_setup()
  File tempest/api/identity/admin/v3/test_endpoints.py, line 51, in resource_setup
    enabled=True)['endpoint']
  File tempest/services/identity/v3/json/endpoints_client.py, line 42, in create_endpoint
    resp, body = self.post('endpoints', post_body)
  File tempest/lib/common/rest_client.py, line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File tempest/lib/common/rest_client.py, line 642, in request
    resp, resp_body)
  File tempest/lib/common/rest_client.py, line 695, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Could not find service: bec7c572cfce4c7687cbfba080313929', u'code': 404, u'title': u'Not Found'}

In this case subunit says that the failed test_id is "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON", not "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON.test_update_endpoint[id-37e8f15e-ee7c-4657-a1e7-f6b61e375eff,smoke]". The whole testcase is not checked by the xfail mechanism so the result is set to normal "fail" and is propagated to the test in it. Finally the test is marked as "fail" not "xfail" in Rally's report .

Changed in rally:
status: New → Triaged
importance: Undecided → Medium
Changed in rally:
assignee: nobody → Yaroslav Lobankov (ylobankov)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/304547
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=a26bc6b2c5b2b9083019292c126416a215391083
Submitter: Jenkins
Branch: master

commit a26bc6b2c5b2b9083019292c126416a215391083
Author: Yaroslav Lobankov <email address hidden>
Date: Tue Apr 12 14:30:47 2016 +0300

    [Verify] Fixing 'xfail' mechanism and whole test case failure issue

    If Tempest fails in a test's setUpClass(), the result that will be reported
    by Rally is "fail", even though the test is listed in a file passed to
    --xfails-file. So this commit fixes the issue.

    Also, this commit adds missing unittest for the subunit parser when we have
    a file with expected failures.

    Closes-Bug: #1568133

    Change-Id: I9e10891f02387b45a507a02326ecf81f0cb97bf3

Changed in rally:
status: In Progress → Fix Released

This issue was fixed in the openstack/rally 0.4.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers