Traces from benchmark context should be gracefull handled

[Artem Panchenko]

Recently I ran rally test (nova boot-and-delete) on OpenStack cloud deployed by Fuel that was running few days, here is the task config:

http://paste.openstack.org/show/117109/

During the test I performed different actions with OpenStack cluster (like adding/removing nodes, simulation of network/hardware failures) which would help to measure how quickly and efficiently HA recovers cloud services under real load. Some of actions caused services downtime and Rally reported, that API returns error, for example:

http://paste.openstack.org/show/117112/

but after API services were recovered it continued to handle requests correctly. In the end of testing (Rally) I turned down some cluster nodes and cloud became unhealthy: HAProxy returned 502/504 errors on all API requests to Nova/Keystone. It caused failure of benchmark tests due to authorization error:

http://paste.openstack.org/show/117108/

and I wasn't able to get tests results (e.g. percent of failed iterations). I think Rally should print error if cluster services become unreachable, but still return tests result even if all requests failed.

I attached full log of tests to the bug (it lacks of beginning of the test, because I enabled logging is screen a little bit later)

[Boris Pavlovic (boris-42)]

During the bunch of various patches this was more or less fixed.

The changes include next:

  - Rally won't fail in case if we get random exceptions from rally benchmark context and runners

  - Rally will LOG these exceptions

  - Rally will put SLA (unexpected_failure) in such case, so you will see that test failed