Replay desyncs

[Toni Förster]

Some replays always desync at the same time-mark and cannot be watched entirely.

REPLAY: Lost synchronization at time 910200
I have: 31cc2400c98213244f817257c9f77e1d
Replay has: 35834365bf28a8d3e50a880deb60cc09

The attached file contains a replay and the corresponding savefile. This particular replay desyncs always at 15min 10sec.

A couple of years ago there was a similar bug reported and fixed
https://bugs.launchpad.net/widelands/+bug/535938/

Part of this bugreport:
https://bugs.launchpad.net/widelands/+bug/1797549/
https://bugs.launchpad.net/widelands/+bug/1797549/comments/13

[Toni Förster (stonerl)]

Very strange, had this issue with another replay. I was able to watch the full 5 hours. Then when I wanted to watch it a second time it always desynced at 18minutes 10. As If the replay file had been changed. I took a look at the time stamps and apparently they were not modified...

[Toni Förster (stonerl)]

Another interesting find. This replay (replay_2.zip) can be watched in its entirety. When I want to watch it a second time, it stops at 18:19, though. But when I close Widelands and reopen it, I can again watch the full replay.

[Toni Förster (stonerl)]

Another finding. The aforementioned replays were created with trunk r8898.

I tried to play them back with r8903.

The first replay (replay-desync.zip) now desyncs at 6min 43sec.
The second replay (replay_2.zip) now always desyncs at 8min 33sec also when played back the first time.

I then tested with r8901. The time were the same as with r8898:

replay-desync.zip:

15min 10sec

replay_2.zip:

first time: unitil the end
second time: 18min 19sec

So some changes between r8901 and r8903 result in different playback behavior.

[GunChleoc (gunchleoc)]

r8901 and r8903 are not compatible regarding replays. We only guarantee replay compatibility per commit to trunk, and r8903 contains a fix to ware transport, so wares are being transported that would heve been lying at a flag forever previously.

[GunChleoc (gunchleoc)]

I can still get a desync for replays with r8934, it I play it at 10x speed. 1x speed is fine.

This is the last output:

MO(6701,thatch_reed): cancel_moving
MO(4321,empire_builder): [transfer]: starting task [movepath] and setting location to road 4077
MO(5680,atlanteans_builder): [transfer]: starting task [movepath] and setting location to road 4049
MO(6883,atlanteans_carrier): [transfer]: set location to road 6881
MO(4238,frisians_builder): [transfer]: starting task [movepath] and setting location to road 4385
MO(4290,frisians_carrier): [gowarehouse]: Got transfer
MO(4290,frisians_carrier): [transfer]: starting task [movepath] and setting location to road 4268
MO(4612,frisians_carrier): [transfer]: starting task [movepath] and setting location to road 6027
MO(4593,barbarians_builder): [transfer]: starting task [movepath] and setting location to road 5362
REPLAY: Lost synchronization at time 925200
I have: a55c218eaf236aba2560ded4e8a0dc26
Replay has: a861bfd2f7cb7768cb6c4958feb53fc1
MO(6866,atlanteans_farmer): [transfer]: starting task [movepath] and setting location to road 4435
REPLAY: Lost synchronization at time 925400
I have: 98c700ec04dc8d695fd8ce6a26e7ae35
Replay has: 8001af54484cdf6fbcb820d94bf5fec3
MO(6935,barbarians_carrier): [transfer]: starting task [movepath] and setting location to road 4179
REPLAY: Lost synchronization at time 925600
I have: 12a53742846b8fed67d849ba6ca61136
Replay has: bb1e7451870457ae58cd8d0e93583ba9

Seems to be related to transportation, so let's test again after https://code.launchpad.net/~widelands-dev/widelands/remove-anti-congestion-algorithm/+merge/359294 has been merged.

[kaputtnik (franku)]

bzr8937[trunk] (Release)

The bug is still there. Played a complete game with this revision. The replay does stop at 5:40 Gametime with gamespeed set to 2 (was impatient to run with 1x speed). Not sure if it is related: The replay stops close after the constructionsite of the empire sentry in the south get connected to the road network.

If widelands is closed at all and restarted, the replay works until the end, although the speed is set to 10x.

The only thing which does not affect playing the replay is, if you start the editor prior to watching the replay. All other cases let the replay stop after 5:40 if you do not close widelands in between:
1. Start a tutorial (and close it as soon as possible), start replay
2. Watch another replay, stop watching, start replay
3. Start a new game, stop it, start replay

Maybe the 'recorder' of replays isn't closed correctly after closing a game? At least it sounds like such to me.

[kaputtnik (franku)]

My observation #7 may was only valid for singleplayer games. The attached multiplayer game does always lost synchronisation after 9 minutes and 06 sec. This happens regardless if the program was restarted or not.

[Teppo Mäenpää (kxq)]

Attached a syncstream from a failed game. Hopefully it is useful.

[Notabilis (notabilis27)]

I was able to figure out what is going wrong in replay_1.zip in #8. I haven't looked at the other replay files yet, but my findings are able to explain the previous observations as well as it can explain (some? all?) network desyncs.

And the problem is ... drum roll ... a global variable. Guess they are considered evil for a reason after all. ;-)
In Widelands, all economies have a unique ID. So each separated building (i.e., no connecting street) is an own economy with an own, unique, ID. To generate these IDs a simple global variable is increased each time a new economy is created (e.g., remove a street so the street network becomes split). Unfortunately, the global variable is never reset to zero. So for the first game (since restarting Widelands) the economy IDs start with zero, for the second game (without restarting Widelands) they might start at 1422 (depending on the number of economies in the first game).
When the game that is recorded in replay_1.zip has been played the economy IDs were around 170 (maybe there has already a multiplayer game been played before?). However, when watching the replay the economies within the replayed game start with ID 0 (or whatever, but most likely not at 170). So when the replay file contains the command to change the economy settings of economy 176 shortly before the time 9:06 the command is ignored since there is no economy with this number. In that case, it was a restriction for wood planks that was reduced from 40 to ~28. By itself, this failed reduction doesn't matter at first. Later on the sawmill checks whether it should work. In the original game, it decided against working since the economy limit was at 28 and there already were 30 planks. In the replayed game however it decides to work since the reduction of the economy target was never executed, resulting in the observed desync.

This can be reproduced by:
1) Start a game and abort it
2) Start a second game
3) Increase the economy target for planks to 50
4) Build a sawmill. It should start working since the economy target is high enough
5) [Optional] Restart the game
6) Watch the replay. It should desync when the sawmill is finished building

A similar sequence is possible for multiplayer games, when one player starts a (singleplayer) game before playing multiplayer, and then the economy settings are modified. By doing so I was able to enforce a multiplayer desync.

To fix this problem we have to reset the economy ID counter before each game. Either we reset the global variable when starting a game or replay, or make the ID counter a member variable of the game class. I would prefer the second, even though it is the bigger code change. I can provide the change myself, but would like a second opinion about how to fix it. It might be required to store this counter in savegames and replays, I am not sure about that one yet.

[kaputtnik (franku)]

What a found! Great investigation and explanation! Many thanks!

Are there some arguments against resetting the counter? Although i 'feel' having it as a member of the game class would be a cleaner solution.

Maybe related: https://bugs.launchpad.net/widelands/+bug/1732765 (see comment #12 and below)

[GunChleoc (gunchleoc)]

You may slap me now, I wrote that code. Economy IDs used to be per player, but that was causing too many issues as well.

Adding a game class increases the cycling dependency between Game and Economy, so I'd rather have the creation of a new game ensure that the economy ID is reset.

[GunChleoc (gunchleoc)]

Fixed in build20-rc1