Comment 17 for bug 1506084

Just my 2c here: The regression test runs in full every time I build a Mac OS X binary for the nightlies, i.e. basically every work day. I do not see any failures there currently.

The reason the tests are flaky and fail on some machines is unfortunately due to the nature of Widelands coding.

I know one issue that is a definite problem: saving does not happen through the commandqueue, but through a side loaded channel that relies on realtime. That means that whenever a script issues a save request, saving will happen sometime soon after that. How much gametime passes depends on the busyness of the computer running the game and external factors - like FPS, delay on reading files and so on. Therefore all the tests contain so much sleep()s to make them less brittle.

Another issue that might be a problem, but I do not know: It could be that the simulation is not fully deterministic. That is just a more generalized version of the problem described in the last paragraph since the sideloading of save makes the game non-deterministic. What I mean here is that a game that runs at 100x for 1 minute might look different than the same game running at 1x for 100 minutes. I do not know if that is the case, ideally it should not be, but I am not sure if we ever made sure that it is/isn't the case.