When the scenario tests were originally written, the main goal was to have a series of tests that could be run on multiple datastores while maintaining the same code base (for the most part). This is in the last stages of being realized so the focus is now on having these tests run in the check/gate on each patchset. Before these been accomplished, each individual test needs to be refactored (i.e. broken into smaller parts) to let it to be 'interleaved' with other tests. This will allow long-running single tests (such as those that create an instance) to be run in parallel, thus reducing the amount of time the entire test suite takes.
It is proposed that this work be broken down into smaller parts. First is a general refactor that will remove the creation of the 'runner' from the group. This will allow the same runner to be passed in to each of the newly refactored 'subgroups' so that state can be preserved between them. Once this is done, each existing group can then be split apart into sections. Using the module tests as an example, the following new groups will probably be needed:
module_create: This handles things like creating/deleting modules, etc. No instances are required for this group
module_instance: This handles applying modules to an instance and also creating a new instance with a module. It *won't* wait for the instance to go active however
module_instance_verify: This waits for the instance to go active and runs any other required tests
module_delete: This handles delete related calls, and will delete the instance created by module_instance. It will not wait for it to 'disappear' however.
module_delete_verify: This will wait for the instance to be completely removed.
Specific tests may require more or less groups, depending on their makeup. The above example should give enough details to describe what the flow should look like.
All new tests should adhere to this 'new' standard so that we can preserve as much of the gate resources as possible.
Once all the tests are refactored, the last step will be to 'interleave' them such that wherever possible, 3 instances are being created at the same time, at all times (any more than 3 seem to cause the gate to fail due to not enough resources).
The comments below can be used to paste reviews of the work being done.
Fix proposed to branch: master
Review: https:/