I recently pushed some code for review, and as a good developer I wanted to run the test suite first to make sure I didn't break anything. Most ran ok, but one test suite kept running for ages before I finally closed the window it was running in. The test in question was «test_check_transportation_works_two_ships», though I guess other test suites could cause this problem too. (And I ran into the same issue with that particulart test suite with an unpatched trunk too)
It would be nice if there was a default timeout threshold where the testsuite would stop, mark the test as timed out and move on to the next test. Most tests seem to run fairly quickly so I guess a reasonable threshold would be 5-10 minutes in case it runs slow for some reason. When tests a marked as timeout a developer may then investigate why the test fails (or of course, mark it as ignored until another developer can look at it).
It would also ensure that a testrun will end at some point, rather than being stuck at a particular test step until the end of time. (Unless we at some point encounter the halting problem :P )
Done some more rethinking regarding the latter part. From what I can see, failures will currently send an in-game message with a description of expected and actual result. So this state seems to also keep a window open without closing. That probably means a naive implementation of a timeout would mark failures as timeouts too. Hm...
I guess I should probably mention that one of the long-term goals with this approach would be to make it possible to run the test suite in a continous integration setup*. I'm hesitant to do so at the moment, because it seems like a failing test or becoming stuck would create a process running forever which would then need to be manually stopped somehow.
Widelands r6945
* I've actually set one up at some time in the past :) It built latest revision with GCC and Clang on I believe up-to-date Debian Sid and Arch Linux. Main reasons I discontinued it was that it was a) I don't have spare hardware to keep it running 24/7 so it lived in a set of virtual machines I ran and b) nothing broke for the time it was running, so it didn't seem high priority.
I agree that this would be desirable. It should be fairly easy to add in regression_test.py - it launches Widelands and the time can be taken then. And if the game does not terminate the test can be failed.
Also, there is a related bug that complains that saving is not deterministic (i.e. when you ask for a save, it will happen whenever) which is the cause of the flakyness in the testsuite (well, at least one cause I know off :) ). It helps if Widelands has the full computer to reduce the flakyness somewhat.