Need pre/post test hooks for tempest

[Vishal Thapar]

Tempest tests are run as part of ODL CSIT. For all ODL CSIT test cases we capture some debug information at end of every test which makes it easier to troubleshoot any failures. However, we can't do the same for tempest tests. Only information we can capture is at end of entire suite. This makes it very difficult to troubleshoot sporadic failures. We can't even confirm if it was a failure to do cleanup correctly.

We need pre/post test hooks in tempest where we can run custom commands/scripts to capture debug information before/after every individual test.

Reference: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-gate-stateful-carbon/378/archives/robot-plugin/log.html.gz#s1-s2

[rajiv (rajiv-kumar)]

More clarity on use case

The problem we have today is that if 1 of these 244 fails, I'd like to capture some information(flows on OVS, data on ODL etc.) at that point. but we don't get control back till entire suite including the teardown/cleanup is done. if we can get a hook to capture information in case of failure, before teardown is done, that is also good enough.

[rajiv (rajiv-kumar)]

I had discusion with tempest guy, he suggested one of the approach is to run each tempest test case individsually and run pre and post hook scripts. It can have a huge maintaince cost, as more test cases get's added or some of them deleted etc, we have to keep track of all of them. For other approaches stay tuned for the updates on this bug :)

[Ghanshyam Mann (ghanshyammann)]

yea from Tempest side, it is very costly to provide pre/post hook. To catch the info i feel debug logs should be there. is there any other information you want to capture than that can be presented in logs?

Depends on actual requirement we can check the best way.

[Ghanshyam Mann (ghanshyammann)]

Got to know we have pause cleanup feature
- second last bullet in New Features

https://docs.openstack.org/releasenotes/tempest/unreleased.html#id6

You need to make this flag true and analyze the data. will it help?

 - https://github.com/openstack/tempest/blob/6c562fbc01263eca890c4f83454a897d83f5bad3/tempest/config.py#L1083

[Andrea Frittoli (andrea-frittoli)]

If you are writing your own tests you can use setUp and tearDown to collect more information.

If this is about existing tests I would like to understand what kind of log you are missing.
It may be something interesting to implement in for OpenStack CI as well.

We try to keep our tests debuggable as much as possible since we run them a lot in the gate and it would be a nightmare otherwise to debug failures.

[Doug Schveninger (ds6901)]

If tempest logging is at debug level the restclient will log all api req- id an you can gather them and look at the server logs to see what is happening. We have a subunit tool that gather them and creates an html report the link in to our kibana instance. Just a suggestion to help to debug when tempest test fails.

https://github.com/openstack/tempest/blob/master/tempest/lib/common/rest_client.py

[Martin Kopec (mkopec)]

removing assignee as there has been no activity for more than a year

[Martin Kopec (mkopec)]

This has been in Incomplete state for several years, it's time to mark this as Invalid.