After a while, increasing game speed makes the game lag even at lower speeds

[Hans Joachim Desserud]

Recently I have noticed that I can increase the speed of games a lot less before they start lagging than before. I am not sure whether the recent performance fixes are to blame, but I noticed it after these were included.

I will attach a replay from Impact with three players (r6063). I'm curious to know whether this affect others or if there is something weird going on on my system.

When I attempt to load it on my machine, I'm able to run it at 30x for a while before it starts lagging, and I have to turn it down to less than 10x. When playing the game, it got so bad that even 2x would lag. This is by the way the second game I noticed this problem in, so it seem to affect all games I play.

I realize that the more buildings and workers are present on the map, the less potential max speed will be, but it used to be a lot more.

For comparison, I loaded the replay from last play day where we played Ancient Sun of Fire in build16. There I was able to keep the speed at 50x until the first players reached the center before it started lagging so bad I had to decrease the speed.

Widelands r6063 on Ubuntu 11.04.

[SirVer (sirver)]

I can reproduce the lag. The culprit is not the new optimizations (or at least I hevily doubt it). The culprit is Map::find_reachable which makes ~20% of CPU on my box. I have no idea if this is related to the blocking stones in the middle or the water area around.

Please do the follwing for more information: recheck a similar game with b16 on the same map (I assume the lag is there as well) AND with the current build, try on another map (like ancient sun of fire). It should be sufficient to let CPU vs CPU.

[SirVer (sirver)]

I removed the regression tag until proven otherwise.

[Hans Joachim Desserud (hjd)]

I created some multiplayer games with AI players and ran them for a while, here are the results.

b16:
ancient sun of fire: 40x for 5 minutes
impact: 65x for 5 minutes (at which point the yellow player taken over the entire map)
Noticed no problems.

r6066:
ancient sun of fire: 30x. After less than a minute I had to decrease the speed to less than 10x.
impact: 30x. After less than a minute I had to decrease the speed, and then I think it froze completely.

In both the r6066 games I noticed lots of messages like 7: building road, immovable in the way, type=7 printed in the terminal, but I don't know if they are related.

>I have no idea if this is related to the blocking stones in the middle or the water area around.
I don't think so. I first noticed it when playing a game on "Three warriors" which is a lot more open.

[SirVer (sirver)]

Okay, I accept the regression tag... I also have an idea where this comes from, but it is not related to the speed improvements but rather to building programs: I noticed that if one fails it is immediately restarted. I guess this could be the regression and I might have introduced it with the stock levels. I'll investigate.

[SirVer (sirver)]

Fixed in r6073. Please keep comparing the new version to b16 if you can. I am very interested in those results.

[Hans Joachim Desserud (hjd)]

Thanks. The main issue is obviously fixed, though I suspect there might be some smaller issues as well remaining.

I tried running some games of Impact on 65x in both build16 and current development version (r6078), and it seems that trunk slows down slightly more especially with lots of soldiers on the screen. Though, the games were not 1:1 comparable, so there might simply have been larger battles going on. I think this needs some more investigation, and I'll look a bit more into it to see if the pattern repeats. Anyways, the remaining problems are in the category "run the game at 50x instead of 60x" so they are in no way as severe as the original one. I also reckon large battles slowing down the game is among the known issues.

[Hans Joachim Desserud (hjd)]

A small update here, since I had some spare time to run some games and look at them.
First of all, I have to admit that for the previous checks I had compiled the latest development version in debug mode, while build16 was compiled in release mode. Turns out there is a large difference when compiling in release mode, which I assume is because it leaves out various debug information. When compiling in release mode wl is able to play at higher speeds without lagging compared to debug mode at least.

Secondly, I switched from testing multiplayer games with all AI players to single player games. Multiplayer games has a maximum speed (65.5x) which makes sense in that context, but is a bit cumbersome for my comparisons. Since single player games doesn't have this cap they are a lot more fit for this purpose.

Ok, so with latest bzr revision compiled in release mode vs build 16, I notice some differences. My method is mainly starting new games on a map, turn up the speed a lot, then when I notice it starts lagging or moving on the map is not as smooth anymore, turn down the speed until it is. At the same time I try to keep an eye on statistics to notice how many hours of game time it is able to keep this up.

I'm happy to say that the development version can cope with higher speeds! I noticed that if I could turn up the speed in build 16 to 150x, I could often set it to 250x (or more) . There seems to be a general rule that it can take the speed from build 16 plus an additional 100, at least initially. I haven't compared how long they are able to keep up these speeds, but they seem roughly the same (after 2 hours in-game things start lagging). Since I am doing this manually, it is also hard to pin point exactly when things start lagging and even harder to notice if the decline in speed is going faster or slower than compared to build16.

One thing I have noticed though is, as mentioned above, battles. Whenever there are bursts or short periods where the game starts lagging on higher speeds, there's attack messages printed in the terminal. By looking at it, this does not seem to affect build16 or at least not to an extent where it is noticable though. So I am not sure whether something has changed making combat slower or whether everything else has improved in general making it the weakest link. I'm afraid I don't know whether it is caused by the combat code directly, or perhaps AI attack/defense calculations happening at the same time, so it would require more investigation to figure out what exactly is causing this.

Note that the tests I have performed are very unscientific, and I haven't really looked at any profilers to get more details about what is going on. (Comment #3 suggests SirVer is using some at least, and any suggestions about tools I could look into are welcome.) It also annoys me a bit that I cannot run older replays in a newer version, which would make it easier to directly compare the same game across different versions. When starting different games in different versions they are not going to be identical and may have subtle (and not-so-subtle) differences with affect the speed. The various choices the AIs can do during a game ma...

[SirVer (sirver)]

Thanks for the detailed feedback. It relieves me that you see a
performance increase - I worked hard for it :). I am using Apples
Instruments for profiling (which is only available for Mac OS X). Other
tools include valgrind which is a beast to work with but very powerful.

[SirVer (sirver)]

Released in build17-rc1.