[OSTF]Need to add retries on each tests method

[Tatyanka]

We have workaround for Oslo-messaging, when it faced with AMQP server on 10.108.20.5:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
it retries to connect after 1-3 seconds several times, but when ostf running, some of test method can invoke openstack exactly in time when oslo-messaging trying to re-connect. As result we can false-negative ostf tests statuses like this one:

http://jenkins-product.srt.mirantis.net:8080/job/fuel_master.ubuntu.bvt_2/310/testReport/junit/%28root%29/deploy_ha_vlan/deploy_ha_vlan/

As soon as env was reverted - I run tests again and it is passed

[Mike Scherbakov (mihgen)]

Tanya, what exactly those test methods are? If it's access to OpenStack REST API, and it fails due to ECONNREFUSED to RabbitMQ - then I would say that it is OpenStack issue: it should handle this exception, and hold users request to REST API instead of responding with error. It can retry to access RabbitMQ, so user will just wait a little bit longer - but will get successful response.

[Tatyanka (tatyana-leontovich)]

Fully agree with you, API calls fails on rabbitmq issue, but after 3 seconds left, rabbitmq works great( it reconnected ) and ostf shows that nothing works. We can leave as is, but there is the chance that such situation will happens. From the other hand, add retry in API calls, will increase test duration on 60 sec( seems it is not too much)

[Mike Scherbakov (mihgen)]

So you confirm that this is actually OpenStack issue. We can make retries for OSTF, yes, if we intend to test OpenStack under normal conditions. But I'm sure that we have to file a bug to OpenStack which should address this issue, and I'd expect that OpenStack'ers will write enough tests to decrease chance of regression in the feature.