Sometimes ware(s) are lying at a flag and get not transported.

[kaputtnik]

Encountered this two times now: Sometimes a ware is lying at a flag and get not transported.

In the attached savegame there is a plank lying at a flag and all carriers of the attached roads do nothing. Very annoying is that the construction site of the mine needs just this plank, see my observations in the following comments.

BZR r8867

[kaputtnik (franku)]

1. Connect another road to the construction site of the mine (eg. Flag at 53/118 to Flag at 49/118): Result situation remains
2. Disconnect road from Flag 51/111 to Flag 51/112: The planks get transported to the construction site.

Disconnecting other 'old' roads to the construction site does not change anything (the planks remain at the flag).

[kaputtnik (franku)]

Sorry the correct position of the flags mentions in 2 is: Disconnect road from Flag 52/111 to Flag 51/112:

Another observation:

Remove road between Flag 52/111 and Flag 51/112 and immediately connect the flags again. Result is that the planks are transported to the flag of the port, but lying around there without transportation.

[kaputtnik (franku)]

Unashamedly assign this bug to ypopezios, because the latest changes to transportation system was made by him :)

[ypopezios (ypopezios)]

Don't count on me anymore, since I lost all motivation in this project.

[GunChleoc (gunchleoc)]

The reason that cutting the road works is that the carrier is twiddling his thumbs - state is taskRoad, but he won't run any of the associated functions.

loader carrier task road
loading state vars 1 0 0 0 (-1, -1)
loader carrier operation -1

[GunChleoc (gunchleoc)]

I have attached a branch with a dirty hack that will periodically unfreeze the idle flags. The savegame has 20 frozen flags in it.

Unfreezing flag for Player 1 at (30, 70) - 1 wares
Unfreezing flag for Player 1 at (52, 24) - 1 wares
Unfreezing flag for Player 1 at (52, 40) - 2 wares
Unfreezing flag for Player 1 at (56, 34) - 3 wares
Unfreezing flag for Player 1 at (56, 39) - 5 wares
Unfreezing flag for Player 1 at (56, 41) - 6 wares
Unfreezing flag for Player 1 at (56, 43) - 1 wares
Unfreezing flag for Player 1 at (68, 37) - 6 wares
Unfreezing flag for Player 1 at (87, 83) - 1 wares
Unfreezing flag for Player 1 at (88, 81) - 6 wares
Unfreezing flag for Player 2 at (18, 130) - 1 wares
Unfreezing flag for Player 2 at (26, 112) - 2 wares
Unfreezing flag for Player 3 at (199, 199) - 6 wares
Unfreezing flag for Player 3 at (205, 214) - 6 wares
Unfreezing flag for Player 3 at (213, 13) - 6 wares
Unfreezing flag for Player 3 at (217, 2) - 4 wares
Unfreezing flag for Player 3 at (218, 7) - 6 wares
Unfreezing flag for Player 3 at (226, 16) - 2 wares
Unfreezing flag for Player 4 at (181, 27) - 1 wares
Unfreezing flag for Player 5 at (52, 111) - 1 wares
Unfreezing flag for Player 5 at (77, 170) - 1 wares

[GunChleoc (gunchleoc)]

I need a break from staring at this. @Notabilis, would you like to have a look?

The attached savegame has another interesting flag at (199, 199). After my unfreezing hack in the attached branch, the carrier will transport 1 ware. Another ware is arriving from the building, which the carrier will also transport. Then the flag is frozen again. I think there is something wrong with the ware not having been routed properly.

[Notabilis (notabilis27)]

Why not, I can take a look at it.

I haven't looked at the other flags, but from a quick look the flag at (199, 199) might be alright. Have you noticed that there is no warehouse in that economy (only a port under construction)? The two stones are moved to two building sites that request them, as soon as the second one is placed by the AI. It might be a bug that the first building site isn't supplied, not sure. Since I was looking at the save in trunk, your unfreezing hack isn't related to the two stones being moved.

Have you pushed all changes to your debug-branch? I tried to use it, but it fails directly after loading saves (either with a failed assert or with ASAN errors).

[Notabilis (notabilis27)]

Playing around a bit found a way to force a ware to get stuck:
- Start a new game
- Start building a Quarry (or whatever) and connect it to the warehouse/HQ
- While the carrier is carrying the ware, set the allowed number of trunks in the construction site to zero
- The ware will now be placed at the next flag and won't be moved again. When opening the debug info for the flag the next destination flag of the trunk is correctly reported on console but no carrier feels responsible
- This can be repeated over and over again in both directions (to construction site / to warehouse):
  - Changing the request while the ware is carried will result in the ware getting stuck
  - Changing the request while the ware is at a flag will work correctly

So it seems something is wrong when re-routing wares. The ware knows where it wants to go but the carrier is not notified of its new schedule. It doesn't matter whether the next carrier that should handle the ware is the one currently carrying it or some other carrier.

[Notabilis (notabilis27)]

I pushed a branch fixing the above mentioned issue. I am not convinced that it is the only issue, though. Removing the road to a construction site does not lead to the same behavior so I am not sure whether it really is the (only) bug from the savegame.
kaputtnik, do you remember whether you were using the "capacity settings" a lot, especially whether you were using it with the mentioned mine construction site?

[GunChleoc (gunchleoc)]

Thanks for looking into this! I'll test your merge request ASAP so that we can get this in to see if the problem is still there in other circumstances.

> Have you pushed all changes to your debug-branch? I tried to use it, but it fails directly after loading saves (either with a failed assert or with ASAN errors).

I always use release builds during development and only switch to debug for the final checks, so I simply didn't run into those bugs in my branch.

[kaputtnik (franku)]

> kaputtnik, do you remember whether you were using the "capacity settings" a lot, especially whether you were using it with the mentioned mine construction site?

Unlikely: The situation of the initial situation was that two already build mines at the same hill got depleted but there was an unconnected construction site of another mine. So i dismantled the two empty mines and connected the construction site of the mine with the road system, so it was build up. Because i needed the mine it is unlikely that i reduced the "capacity settings" of logs for this mine. But i am not sure if a played around with the "capacity settings" of logs of the port(s) and/or other warehouses. Also i can't remember if i had dis/reconnected some related flags of the related mines.

Maybe it wood be good having a console output "Idle ware at flag x/y found!" or similar in debug builds. If possible with the destination of that ware?

[GunChleoc (gunchleoc)]

I have merged the fix into my dirty hack branch to make it crash when we get an idle ware, but I am getting a bad_cast somewhere. So, that needs to be fixed before we can use it for that kind of testing.

[Notabilis (notabilis27)]

I tested some more and the same bug can also be triggered by dismantling buildings. So in general it can appear when a ware is rerouted (for whatever reason) and the carrier would have to carry it back. I can't say where exactly your two mines have been standing but I guess it could have happened in your case as well (log is on route to the warehouse after dismantling but then rerouted to the construction site). So maybe that bug really is the only reason for all the stuck wares, seems much more likely now.
On a side note, I noticed that my fix falsely triggers sometimes. I don't think that leads to any problems for the economy (the second call to the functions will result in the same "nothing to do" result as before), only costs

I guess it would be possible to add a console output on idle wares. We could also add some code to the flag which periodically wakes up the idle wares stored there, similar to GunChleoc's hack (but more efficient I think). I don't know whether either is needed, though.
Most stuck wares will get unstuck after some (probably short) time (by some other ware motivating the lazy carrier to do something) so we would end up with quite a lot of false positives. That is probably no problem for the "wake-up" code but might confuse people looking at the logs.

[kaputtnik (franku)]

> I tested some more and the same bug can also be triggered by dismantling buildings.

That is what i 'felt' to trigger the bug, but couldn't reproduce it this way.

> Most stuck wares will get unstuck after some (probably short) time

Yes, that happened when i first discovered this bug. That time it was a piece of wheat that lay at a flag without transportation, but after some time it get picked up by a carrier and found his way to the destination.

> so we would end up with quite a lot of false positives.

I think each stuck ware is a bug, regardless if it will be freed after some time, even if the time is short. If a player is short on some ware and that ware is stuck at a flag, he has a big disadvantage.

> That is probably no problem for the "wake-up" code but might confuse people looking at the logs.

I guess only people interested in bug hunting will look at the logs. So having such a log would be helpful i think.

[GunChleoc (gunchleoc)]

I have added a branch "flag_unfreezer" that will trigger an exception when a flag is frozen for a long time. We can use this to run a couple of test games - I have already managed to have it trigger once, but when I reloaded the game to get a better autosave, the AIs changed their decisions and the bug didn't happen again. So, no autosave to analyze yet.

[GunChleoc (gunchleoc)]

Using the flag_unfreezer branch, I got a hit with the attached savegames. If I load the oldest of these autosaves, I can get a hit again, but not necessarily with the same flag.

[GunChleoc (gunchleoc)]

Possibly related: https://wl.widelands.org/forum/topic/4381/

[kaputtnik (franku)]

Would it be real problem if the changes which probably causes this bug(s) get reverted?

The initial problem of a congestion wasn't ever reported as a problem, and the issue with AI road congestion may can (should) be solved on the AI level, imho.

Ai roads get jammed: https://bugs.launchpad.net/widelands/+bug/1535115

[GunChleoc (gunchleoc)]

The reason I am trying to keep this in is because the commit broke savegame compatibility and Tibor will have to create all the savegames for his AI training setup again.

If we end up fixing everything except for this bug, we can still revert the changes then.

[GunChleoc (gunchleoc)]

There is another bug where carriers have an endless walking cycle trying to enter a warehouse.

I'll remove the algorithm for now, we can always raid it for ideas after Build 20.

[GunChleoc (gunchleoc)]

Fixed in build20-rc1