Convergence tests are slow

[Steven Hardy]

Slowest Tests
Test id Runtime (s)
---------------------------------------------------------------------------------------------- -----------
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_replace_missed_cleanup) 7.089
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_user_replace) 6.373
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(multiple_update) 5.947
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_replace) 5.896
heat.tests.test_environment.ResourceRegistryTest.test_list_type_with_is_available_exception 5.397
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_user_replace_rollback) 5.133
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_replace_rollback) 4.966
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(basic_update_delete) 4.158
heat.tests.convergence.test_converge.ScenarioTest.test_scenario(update_add_rollback_early) 4.112
heat.tests.db.test_migrations.TestHeatMigrationsSQLite.test_walk_versions 4.012

We're spending about 45seconds running only 8 convergence scenario tests, which is a big overhead when you run the tests regularly.

I assume we're missing some stubbing of sleeps, or perhaps these scenario tests should only be run as functional tests, not the default tox -e py27 target?

[Rakesh H S (rh-s)]

These are scenario tests for convergence code(https://review.openstack.org/#/c/203661/).
It acts on fake_resources only.

It could be moved to functional tests, but since it takes less than a minute and till the time all convergence bugs are fixed and convergence is made the default option, I think it is fine to have it here so that dev will uncover bugs in early stage. (I assume not everyone runs the actual integration tests before submitting code)

Regards,
Rakesh

[Rabi Mishra (rabi)]

IMO, we've voting convergence gate job to uncover any bugs. I would be +1 to move them to the functional tests and changed to use actual resources.

if unit test coverage is not there for convergence code, we should add them.

[Steven Hardy (shardy)]

FWIW, I think these are functional tests, and should not be run via the default tox -epy27 target - they should be run in the gate, and as Rabi says, this will ensure we maintain coverage, but these aren't unit tests AFAICT.

[Zane Bitter (zaneb)]

Last time I checked we have no tests >3s, and the scenario tests don't appear in the slowest list. So I think we can call this solved.

[Zane Bitter (zaneb)]

Also these are definitely not functional tests. They rely on stubbing out a good portion of the engine, and the code is designed to be shared with https://github.com/zaneb/heat-convergence-prototype