If Tempest fails in a test's setUpClass(), the result that will be reported by Rally is "fail", even though the test is listed in a file passed to --xfails-file.
In my case, I deployed Keystone with the "templated" catalog driver. This driver, unlike the "sql" driver, doesn't support APIs for services or endpoints. So the test "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON.test_update_endpoint[id-37e8f15e-ee7c-4657-a1e7-f6b61e375eff,smoke]" is expected to fail. Actually it does, but the failure occurs in setUpClass() as you can see in the traceback:
Traceback (most recent call last):
File tempest/test.py, line 280, in setUpClass
six.reraise(etype, value, trace)
File tempest/test.py, line 273, in setUpClass
cls.resource_setup()
File tempest/api/identity/admin/v3/test_endpoints.py, line 51, in resource_setup
enabled=True)['endpoint']
File tempest/services/identity/v3/json/endpoints_client.py, line 42, in create_endpoint
resp, body = self.post('endpoints', post_body)
File tempest/lib/common/rest_client.py, line 259, in post
return self.request('POST', url, extra_headers, headers, body)
File tempest/lib/common/rest_client.py, line 642, in request
resp, resp_body)
File tempest/lib/common/rest_client.py, line 695, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Could not find service: bec7c572cfce4c7687cbfba080313929', u'code': 404, u'title': u'Not Found'}
In this case subunit says that the failed test_id is "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON", not "tempest.api.identity.admin.v3.test_endpoints.EndPointsTestJSON.test_update_endpoint[id-37e8f15e-ee7c-4657-a1e7-f6b61e375eff,smoke]". The whole testcase is not checked by the xfail mechanism so the result is set to normal "fail" and is propagated to the test in it. Finally the test is marked as "fail" not "xfail" in Rally's report .
https:/