Implementing a relay server

[Notabilis]

Two short notices before you continue reading:
- This bug report is more like a project proposal/description but launchpad blueprints do not seem to have a way to discuss them.
- The following plans are concerned with the underlying TCP connections of the network gaming and do not affect the gameplay as such.

Currently, when hosting an internet game of widelands the host player has to create port forwarding rules in his router. For a lot of players this is a problem. As a possible solution the introduction of a relay server was proposed quite some time ago [1]. The basic idea is that all players (even the host) are connecting to this globally reachable relay so they don't require any port forwardings on their own systems.
[1] https://blueprints.launchpad.net/widelands/+spec/metaserver-game-communication-module

Another older request is support for IPv6 [2], which will probably become even more pressing in the future. Since this seems to require a change of the used networking library I would prefer to do it before starting to implement a relay server so we can use the same technology for both.
[2] https://bugs.launchpad.net/widelands/+bug/1092567

In the following I will describe what I am currently planning to do. If you spot any problems or have other comments, feel free to tell me.

== Refactoring Step 1 ==
I renamed NetHost/NetClient to GameHost/GameClient and am working on extracting the network specific parts into new NetHost/NetClient classes (no branch online yet). The plan is to have all network calls hidden so they can be easily replaced.

== Using Boost.Asio ==
When the classes are extracted, I intend to replace SDL_net with Boost.Asio since there is no IPv6 for SDL_net (only an unofficial modification for an outdated version of SDL_net).

== IPv6 ==
When we already use Boost.Asio, it should be quite easy to add support for IPv6 in the relevant wideland classes. I haven't tried it out yet but I think it should even be possible to support IPv4 *and* IPv6 clients in the same game without any special handling (as long as the host-computer supports both protocols, of course).

Metaserver:
The metaserver seems to already support IPv6 based on a quick view through its code. However, the domain name widelands.org can not be resolved for IPv6 so I have no idea whether the metaserver-computer itself supports it yet. If someone could enable it that would be great. No idea whether there are any firewall rules which have to be updated, too. Note that this would probably also affect the homepage.
Currently the metaserver reports the IP address the game host used to connect with it. In the future it might be worth changing that to allow IPv4/IPv6 participants in the same game.

== Refactoring Step 2: Proxies Part 1 ==
Since the GameHost does no longer have direct access to the network functions, we can replace the underlying network structure. The idea is that the NetHost class will be replaced by a NetProxy class and a NetRelay (or so).
The NetRelay is started on the same computer and the NetProxy as well as the clients connect to it. Its task is to simply relay all traffic between the NetClients and the NetProxy. The GameClient and NetClient won't notice any difference compared to talking to the host directly. Since the NetHost and NetProxy classes will offer the same interface, the GameHost shouldn't notice either. Between the NetProxy and the NetRelay the connection will be multiplexed with some simple protocol to allow host messages to/from all/single clients. The clients are only communicating with the host (over the relay) so there is no change required.

Admittedly always using the relay is a small overhead compared to using the (removed) NetHost when possible. But it is less code that way so I decided in favor of it.

== Proxies Part 2 ==
With the separation of NetProxy and NetRelay the NetRelay can also be made available as a separate program to run on the computer of the metaserver. The idea is that the metaserver tries to connect to the relay offered by the host. When it works, fine, continue as we do now. If it doesn't, start an own NetRelay on the metaserver and tell the GameHost about it so it connects there. The clients will also be told to connect to the NetRelay, so in that case the game will be relayed over the metaserver.

Note that until here the changes are all locally and should be compatible with the netcode of the current trunk. If I am not overlooking something it should be possible for trunk-clients to connect to the relay on the metaserver without ever noticing it.

== Proxies Part 3 (crazy ideas) ==
The NetRelay does not necessarily has to run on the metaserver. For example, it could run on the computer of any participant in the game (even observers), reducing the load on the metaserver. The latency would not necessarily become worse since it doesn't matters whether we have the metaserver or some client as an additional hop (besides: We don't have much data anyway).

We would have to teach the client about the relay when doing so, though. But then we could even go as far as changing the relay while ingame, when e.g. an observer or a loosing player was hosting the relay and he disconnects. (For that case we would have to add our own segment numbers to packets to avoid loosing any when switching relays.)

== Further work ==
- When the metaserver code is touched anyway we should probably add support for encrypted password transmission on login. The easiest way would something like CHAP. Or we go for a completely encrypted metaserver connection with TLS (which would mean a new library requirement in form of openssl for the clients).

- Currently, when a connection is lost the game has to be re-hosted and a savegame loaded. It would be comfortable if the disconnected client could simply reconnect to the server even when his IP changed. Shouldn't be that difficult since the server is waiting for the client and both have stored the same gamestate anyway. This probably even works when the server gets lost shortly as long as he tells the metaserver his new IP and the clients get it from there.

- Late joins of observers would be a nice feature. In the tournament it happened that potential observer missed the start of the match by just a few minutes and couldn't join later on. Not sure how much work this one is since we have to transmit the game state to the new observer, preferably without disturbing the running game.

So much for my plans for the next ten years. Any comments, critic or praise are welcome. If no-one protests I will just start working on it and sooner or later there will be some merge requests.

[SirVer (sirver)]

Peter and I also played back and forth on a possible design here https://docs.google.com/document/d/1hwGSytvJRjCIcndAk3dR18ZCtEo_5eaiykzBSKw2krQ/edit?disco=AAAAAi1Xro4.

I am very, very much in favor of this proposal - no matter how it will look in the end. My thought was to add this to the metaserver, but I like the always-use-proxy even better. I thought I would eventually implement this, but it seems my time for Widelands is quite limited :(.

[Notabilis (notabilis27)]

I'm glad that you approve. Maybe you can help me with a piece of the current networking code. At multiple places, e.g. in network/SDLNet_ResolveHost.cc after line 100 [1], we are resolving the hostname by gethostbyname() only to overwrite the result again a few lines later by a call to SDLNet_ResolveHost().
The only reason I can see is that someone used gethostbyname() to check whether the resolution is possible at all. This can also be done by checking the return value of SDLNet_ResolveHost(), but maybe that was overlooked? If you don't have any idea either I will drop the gethostbyname() call.

[1] https://bazaar.launchpad.net/~widelands-dev/widelands/trunk/view/head:/src/network/internet_gaming.cc#L100

[SirVer (sirver)]

No idea why that is used. Did you look at the blame history for this line?

> Am 08.05.2017 um 08:58 schrieb Notabilis <email address hidden>:
>
> I'm glad that you approve. Maybe you can help me with a piece of the current networking code. At multiple places, e.g. in network/SDLNet_ResolveHost.cc after line 100 [1], we are resolving the hostname by gethostbyname() only to overwrite the result again a few lines later by a call to SDLNet_ResolveHost().
> The only reason I can see is that someone used gethostbyname() to check whether the resolution is possible at all. This can also be done by checking the return value of SDLNet_ResolveHost(), but maybe that was overlooked? If you don't have any idea either I will drop the gethostbyname() call.
>
> [1] https://bazaar.launchpad.net/~widelands-
> dev/widelands/trunk/view/head:/src/network/internet_gaming.cc#L100
>
> --
> You received this bug notification because you are subscribed to
> widelands.
> https://bugs.launchpad.net/bugs/1689087
>
> Title:
> Implementing a relay server
>
> Status in widelands:
> Confirmed
>
> Bug description:
> Two short notices before you continue reading:
> - This bug report is more like a project proposal/description but launchpad blueprints do not seem to have a way to discuss them.
> - The following plans are concerned with the underlying TCP connections of the network gaming and do not affect the gameplay as such.
>
> Currently, when hosting an internet game of widelands the host player has to create port forwarding rules in his router. For a lot of players this is a problem. As a possible solution the introduction of a relay server was proposed quite some time ago [1]. The basic idea is that all players (even the host) are connecting to this globally reachable relay so they don't require any port forwardings on their own systems.
> [1] https://blueprints.launchpad.net/widelands/+spec/metaserver-game-communication-module
>
> Another older request is support for IPv6 [2], which will probably become even more pressing in the future. Since this seems to require a change of the used networking library I would prefer to do it before starting to implement a relay server so we can use the same technology for both.
> [2] https://bugs.launchpad.net/widelands/+bug/1092567
>
> In the following I will describe what I am currently planning to do.
> If you spot any problems or have other comments, feel free to tell me.
>
>
> == Refactoring Step 1 ==
> I renamed NetHost/NetClient to GameHost/GameClient and am working on extracting the network specific parts into new NetHost/NetClient classes (no branch online yet). The plan is to have all network calls hidden so they can be easily replaced.
>
> == Using Boost.Asio ==
> When the classes are extracted, I intend to replace SDL_net with Boost.Asio since there is no IPv6 for SDL_net (only an unofficial modification for an outdated version of SDL_net).
>
> == IPv6 ==
> When we already use Boost.Asio, it should be quite easy to add support for IPv6 in the relevant wideland classes. I haven't tried it out yet but I think it should even be possible to support IPv4 *and* IPv6 clients in the same game wi...

[GunChleoc (gunchleoc)]

I did some digging while waiting for the compiler.

gethostbyname is deprecated for Windows:

> If the host specified in the name parameter has both IPv4 and IPv6 addresses, only the IPv4 addresses will be returned. The gethostbyname function can only return IPv4 addresses for the name parameter. The getaddrinfo function and associated addrinfo structure should be used if IPv6 addresses for the host are required or if both IPv4 and IPv6 addresses for the host are required.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms738524(v=vs.85).aspx

And for Linux as well:

> The gethostbyname*() and gethostbyaddr*() functions are obsolete. Applications should use getaddrinfo(3) and getnameinfo(3) instead.

https://linux.die.net/man/3/gethostbyname

SDL2_net-2.0.1 still uses gethostbyname only, so your idea with Boost.Asio is a good one.

[Notabilis (notabilis27)]

I traced that occurrence back to a commit [1] by Nasenbaer when he removed the ggz usage. It already appears in basically the version as it is today.
Since you two seem to agree with me that it does not make sense (at least not anymore), I will remove it.

[1] https://bazaar.launchpad.net/~widelands-dev/widelands/ggz-replacement/view/6218/src/network/internet_gaming.cc

[GunChleoc (gunchleoc)]

I just stumbled across a patch hat 0AD are working on to eliminate the need for port forwarding. I don't know if it's useful to us, but here it is: https://code.wildfiregames.com/D364

[Notabilis (notabilis27)]

Thanks for the pointer, I haven't considered that approach (mainly because there already were ideas about a relay server, I think).

The problem we are having is that connections incoming from a computer over the internet to local hosts are blocked by NAT or by firewalls. The reverse direction is normally permitted: A local host can have outgoing connections to any server in the internet. So we need some way to punch through the NAT/firewall from the outside to reach the locally hosted game.

We are intending to set up a relay server which can be reached by outgoing connections. Since those are always possible (okay, maybe not in some company networks or so), the clients and the game host can connect to the relay. The disadvantage is, that we have to provide such a server and the network/computational load of forwarding packets.

Their approach (well, the approach they are planing) does not use a relay server. They also need a special server for it, but only to transfer some initial messages between the participants.
The idea is that the host tries to connect to the client, which will most likely fail. But by doing so the host "punches holes" into his NAT since the NAT has to allow answering packets from the client (which won't come in this case, but the NAT doesn't know this). Now the client tries to connect to the server, which will now work since we punched the holes, making the server reachable.
So much for the nice theory. Unfortunately, NATs often change the port of the outgoing network packet. For example, your computer might send a message to a webserver and thinks that he is sending from port 51253. But since the NAT changes the port,the webserver sees another port, e.g., 55742. The same happens in our host/client combination. Here it is even more fun, since this happens on both sides of the intended connection.
That is where the special server comes in. Both host and client connect to it and the server tells them which port he sees the other is using respectively. So when the client tries to connect to the host, he is not using our normal widelands port for it but instead some other port the NAT selected. And now comes the bad part where it breaks: For each new connection (first connection: host to special server, second: host to client) the NAT selects another port. So the client can not connect to the port the special server is seeing but has to connect to some other port. Some NATs behave nicely and select easily guessable ports, e.g., simply increasing. Others are using random mappings so there is no chance that the client guesses the right one.
Compared to the relay server, this approach has the advantage of less load on our servers. The disadvantage however is, that there is no guaranty that a connection can be created.

They already seem to use UPnP as another approach. With UPnP it is possible to automatically establish a port forwarding in the router, which would solve the reachability problem. Unfortunately, this is not enabled in all routers so it won't always work neither.

So we could follow (one of) their approaches but when we want to guarantee a connection we have to offer a relay server anyway. At least, that is how...

[SirVer (sirver)]

> The disadvantage is, that we have to provide such a server and the network/computational load of forwarding packets.

To put some numbers on this: we will need to precisely submit the content of the *.wrpl files to all players that are connected + maybe the savegame that is used to start the game from. All in all this is < 20MB per game. No concern for the traffic.

As for the CPU: we only need to relay, not simulate, so the CPU usage is probably neglectable. However, our V-Host only has a single CPU, so if many games are running in parallel, some cronjob runs and the website sees traffic we might feel it. I expect the extra CPU to be so little that it can be neglected.

> Of course, we can offer a relay server and one of the other approaches, too, in the hope of reducing the load on our server.

I argue that this is not worth the cost of implementation until the load on the server rises. Right now we at max have 3 games in parallel which we should easily handle. Also, as you point out none of their approaches is actually as reliable as a full proxy.

I also like that the proxy gives us a chance of saving and analyzing replays for crashes and strategies.

[GunChleoc (gunchleoc)]

Another question: Is there currently a difference in the code for LAN and internet games? LAN games should still be possible without using an internet connection.

[Notabilis (notabilis27)]

Thanks for the numbers SirVer. So we will just try it and see where it goes.

> I also like that the proxy gives us a chance of saving and analyzing replays for crashes and strategies.
I am a bit worried about this comment. Do you intend to save all games which go over the relay?

LAN and internet games are more or less the same codewise. The GameClient class receives the host it should connect to on construction. Whether that is an internet host or a LAN host does not matter. Depending on what the user does in the main menu we are previously either connecting to the metaserver or listening on the local network. Both sources can provide the network address to connect to for gaming.
Hosting a game is done by the GameHost class. If we started it over the internet lobby we announce it on the metaserver, otherwise its a LAN only game.
So no, LAN games don't and won't need an active internet connection.

[SirVer (sirver)]

> I am a bit worried about this comment. Do you intend to save all games which go over the relay?

I think this is up for discussion. It is what commercial companies do (for example Blizzard has a replay for every single starcraft 2 game ever played, so does riot for league of legend IMHO). It would allow us to do easier bug triage - because we would see more bugs, even auto file them if desired and have relevant triage data at our fingertips. It would also build a database of rare test cases over time.

It would also be nice for tournaments - replays are always there and do not need to be send in.

We have to think this through and discuss this broadly with respect to our users privacy. For example chat messages are private information and must not be stored on the server under any circumstances. What else? Do we scrub user names (i.e. SirVer becomes "Blue player", Notabilis "Red Player")? Do we not keep any replays at all?

I think the initial implementation of the proxy should not store anything. This can be added later - I just think it is worthwhile: it adds a lot of technical merit and the privacy implications seem very small. As said, this should be discussed more broadly in the community beforehand.

[GunChleoc (gunchleoc)]

We could also make this opt-in, so the users can decide themselves which amount of data collection they consent to.

[Notabilis (notabilis27)]

Okay, thanks for the clarification. When we do it as opt-in and with some privacy protection, I don't have any more complains.

[Notabilis (notabilis27)]

Now that boost and IPv6 support is "done" (as in: coded but not reviewed) for non-metaserver games, I am working on adding IPv6 support to the metaserver.

COdewise, it already supports IPv6 and IPv6-only games can be played over it. However, it is currently far from optimal. When the game is reachable by the metaserver by any protocol it is shown as reachable for all clients, even when they are using the other protocol version (making the host not reachable after all). Also, clients can only connect with one protocol to the metaserver.

My current planning is as follows:
- Clients try to connect to the metaserver with both protocols in parallel. The second connection will be closed and only its IP-address will be stored.
- For registered users, both connections do a separate login on the metaserver. The newest connection for each protocol replaces possibly existing old ones (also avoiding the "you are already logged in" messages). Old connections will be closed.
- For unregistered users, the client will send some nonce ( = random value) with his login requests. Two connections will be considered belonging to the same user when username and nonce are matching.*
- Now the metaserver knows which user supports which protocol versions. When a client opens a game, the metaserver can probe both protocols whether the game is reachable.
- Depending on the result, it can send different "you can reach this game" messages to the connected clients, depending on which protocol they and the host support.

If you have comments or see any problems, please tell me.

A (for me unsolvable) problem is that widelands.org only supports IPv4 at the moment. SirVer, are you able and willing to enable IPv6 there? Its not urgent, though, testing works fine locally.

* We could probably permanently store the nonce locally in a config file. This way even unregistered clients can reconnect with their last used name when they are disconnected due to, e.g., network failures. Since each user has an own nonce, stealing names from already logged in users will not be possible.

[GunChleoc (gunchleoc)]

What the server currently does is to add a sequential number to the nickname if the name is reserved or currently in use. So, I don't think that we need an extra random nonce here. Even unregistered users still need to provide a nickname when logging in.

[Notabilis (notabilis27)]

The problem is, that I actually want to have two connections with the same nickname. Normally, this is avoided by the metaserver by appending a number. In my case however, I want to have two connections belonging to the same user, so the metaserver needs to see something he can recognize the user on.
I think only using the nickname would work most of the time. However, It would be easy to assign an arbitrary IP address to any user which an attacker could use to hamper hosting games.

[SirVer (sirver)]

Notabilis, I think https://github.com/widelands/widelands_metaserver is ahead of the lp:widelands-metaserver, but I am not positive. I remember vaguely that we wanted to try out the github workflow with it - but then not too much work happened.

[Notabilis (notabilis27)]

They both have "Merged tinos IRC branch." on the same day as their newest commit, so I guess they are at the same state. I can also request a merge on github if you want. What is the workflow there? Create an own account, clone the metaserver into it and create a pull request?

[SirVer (sirver)]

Okay, as long as you are aware. I think we should probably retire one of the two projects (launchpad or github), but I leave it to Gun which one

> I can also request a merge on github if you want. What is the workflow there? Create an own account, clone the metaserver into it and create a pull request?

More or less. Github has this documentation for this: https://guides.github.com/introduction/flow/

[Notabilis (notabilis27)]

Okay, thanks. Than I will continue working on launchpad until I hear otherwise.

Another problem (not related to this): For the IPv6-change I increased the protocol version of the metaserver protocol. I will also increase the number again for future changes (proxy, protected passwords). Do we keep support for all protocol versions in the metaserver or e.g. just for the newest one and the last stable?

[GunChleoc (gunchleoc)]

Maybe this would be a good candidate for GitHub, because it's not very active, and will be one less project to migrate. However, since you already have a merge request up, take your pick :)

[Notabilis (notabilis27)]

I also started a pull request on GitHub now (mainly to try it out). The workflow is different from launchpad, but it seems to be okay, too.

https://github.com/widelands/widelands_metaserver/pull/2

[SirVer (sirver)]

I reviewed this on GitHub. Nice work! I hope that we can delete the IPv4 code path one day.

> Do we keep support for all protocol versions in the metaserver or e.g. just for the newest one and the last stable?

My 2c: We have to keep support for all stable versions that are still used. We can keep logs of how often old builds still log into the metaserver and decide accordingly.

As for development: keeping support for the stable versions and the trunk seems sufficient to me. so should you add more features before b20 and increase the version again, we can remove support for the intermediate versions. We should keep track in the code which Widelands stable uses which metaserver protocol number though.

[Notabilis (notabilis27)]

Thanks for the reviews! Should be all done.

I started a list of protocol versions in internet_gaming_protocol.h. Quite empty as of now but it will grow.

[Notabilis (notabilis27)]

Now that IPv6 support is done, I started working on the relay server. Ironically, I have an IPv6-related problem with the design. Given the following scenario: One player has only IPv4 and one only IPv6, but they would like to play.

Currently, either of them can open up a game but it won't be connectable by the other player. Now is the question how we want to handle this with the relay server:
1) Ignore it. The game is reachable by at least one of IPv4/6, so everything is fine.
2) Always run games over a relay server on the metaserver computer. (Maybe with the exception of game hosts which are reachable by IPv4 and IPv6.) This would result in higher load on our server.
3) Add the relay server dynamically. Start with a local one (e.g. IPv4 only) and switch to a relay server on the metaserver as soon as an IPv6-only player wants to join. (This would also be a step to "Proxies Part 3" in the first post. In case if anyone is interested in part 3 anyway.)

Any preferences?

[SirVer (sirver)]

I'd go for 2 until we see Problems with this approach. My expectation is that the relay server will have pretty low CPU requirements, so we probably do not care unless Widelands spikes in popularity.

In the long run this should also simplify the design: we do not need to Ping game hosters anymore and games can survive network drops of players including the hosting one better.

> Am 02.07.2017 um 17:15 schrieb Notabilis <email address hidden>:
>
> Now that IPv6 support is done, I started working on the relay server.
> Ironically, I have an IPv6-related problem with the design. Given the
> following scenario: One player has only IPv4 and one only IPv6, but they
> would like to play.
>
> Currently, either of them can open up a game but it won't be connectable by the other player. Now is the question how we want to handle this with the relay server:
> 1) Ignore it. The game is reachable by at least one of IPv4/6, so everything is fine.
> 2) Always run games over a relay server on the metaserver computer. (Maybe with the exception of game hosts which are reachable by IPv4 and IPv6.) This would result in higher load on our server.
> 3) Add the relay server dynamically. Start with a local one (e.g. IPv4 only) and switch to a relay server on the metaserver as soon as an IPv6-only player wants to join. (This would also be a step to "Proxies Part 3" in the first post. In case if anyone is interested in part 3 anyway.)
>
> Any preferences?
>
> --
> You received this bug notification because you are subscribed to
> widelands.
> https://bugs.launchpad.net/bugs/1689087
>
> Title:
> Implementing a relay server
>
> Status in widelands:
> Confirmed
>
> Bug description:
> Two short notices before you continue reading:
> - This bug report is more like a project proposal/description but launchpad blueprints do not seem to have a way to discuss them.
> - The following plans are concerned with the underlying TCP connections of the network gaming and do not affect the gameplay as such.
>
> Currently, when hosting an internet game of widelands the host player has to create port forwarding rules in his router. For a lot of players this is a problem. As a possible solution the introduction of a relay server was proposed quite some time ago [1]. The basic idea is that all players (even the host) are connecting to this globally reachable relay so they don't require any port forwardings on their own systems.
> [1] https://blueprints.launchpad.net/widelands/+spec/metaserver-game-communication-module
>
> Another older request is support for IPv6 [2], which will probably become even more pressing in the future. Since this seems to require a change of the used networking library I would prefer to do it before starting to implement a relay server so we can use the same technology for both.
> [2] https://bugs.launchpad.net/widelands/+bug/1092567
>
> In the following I will describe what I am currently planning to do.
> If you spot any problems or have other comments, feel free to tell me.
>
>
> == Refactoring Step 1 ==
> I renamed NetHost/NetClient to GameHost/GameClient and am working on extracting the network specific parts into new NetHost...

[Notabilis (notabilis27)]

So simply always start a relay server for internet games, independent of whether the game host is reachable?

[SirVer (sirver)]

Yup.

> Am 02.07.2017 um 20:01 schrieb Notabilis <email address hidden>:
>
> So simply always start a relay server for internet games, independent of
> whether the game host is reachable?
>
> --
> You received this bug notification because you are subscribed to
> widelands.
> https://bugs.launchpad.net/bugs/1689087
>
> Title:
> Implementing a relay server
>
> Status in widelands:
> Confirmed
>
> Bug description:
> Two short notices before you continue reading:
> - This bug report is more like a project proposal/description but launchpad blueprints do not seem to have a way to discuss them.
> - The following plans are concerned with the underlying TCP connections of the network gaming and do not affect the gameplay as such.
>
> Currently, when hosting an internet game of widelands the host player has to create port forwarding rules in his router. For a lot of players this is a problem. As a possible solution the introduction of a relay server was proposed quite some time ago [1]. The basic idea is that all players (even the host) are connecting to this globally reachable relay so they don't require any port forwardings on their own systems.
> [1] https://blueprints.launchpad.net/widelands/+spec/metaserver-game-communication-module
>
> Another older request is support for IPv6 [2], which will probably become even more pressing in the future. Since this seems to require a change of the used networking library I would prefer to do it before starting to implement a relay server so we can use the same technology for both.
> [2] https://bugs.launchpad.net/widelands/+bug/1092567
>
> In the following I will describe what I am currently planning to do.
> If you spot any problems or have other comments, feel free to tell me.
>
>
> == Refactoring Step 1 ==
> I renamed NetHost/NetClient to GameHost/GameClient and am working on extracting the network specific parts into new NetHost/NetClient classes (no branch online yet). The plan is to have all network calls hidden so they can be easily replaced.
>
> == Using Boost.Asio ==
> When the classes are extracted, I intend to replace SDL_net with Boost.Asio since there is no IPv6 for SDL_net (only an unofficial modification for an outdated version of SDL_net).
>
> == IPv6 ==
> When we already use Boost.Asio, it should be quite easy to add support for IPv6 in the relevant wideland classes. I haven't tried it out yet but I think it should even be possible to support IPv4 *and* IPv6 clients in the same game without any special handling (as long as the host-computer supports both protocols, of course).
>
> Metaserver:
> The metaserver seems to already support IPv6 based on a quick view through its code. However, the domain name widelands.org can not be resolved for IPv6 so I have no idea whether the metaserver-computer itself supports it yet. If someone could enable it that would be great. No idea whether there are any firewall rules which have to be updated, too. Note that this would probably also affect the homepage.
> Currently the metaserver reports the IP address the game host used to connect with it. In the future it might be worth changing that to allow IPv4/IPv6 parti...

[Notabilis (notabilis27)]

Okay, will do so. Will the relay still be implemented in C++ as part of the game code? My plan was to do so and add an extra "relay-server" project output for an executable for the metaserver. This way, we don't need special code for handling LAN games (just start a local relay).
As an alternative, we could implement it somewhere else (maybe in Go directly as part of the metaserver?). Than we would have to keep the current hosting code in the game for LAN games and add new code for internet games.

Currently, I am quite unsure which way is to prefer. Until now it more or less had to be the first one and it does keep the option open to move the relay server to the systems of the players. But the second one does not sound that bad either.

And just a random thought: I haven't looked yet, but since the logic of the relay server is completely independent of Widelands, we could probably use an existing solution. Any opinions about that?

[SirVer (sirver)]

I read a few open questions in your post:

1) which language should the relay be in?
2) should it be part of widelands, part of the metaserver or a separate binary?
3) Can we reuse something existing?

About 1) The relay will be running on the server and is a potential attack vector - its opening ports. I'd much rather have it written in a safe language (Go, Rust, Python) than C++. Actually, I think this is a hard requirement: a C++ binary would reuse our own network code and I have 0 confidence that it does not contain exploitable buffer overruns. A safe language is much harder to hack.

The go metaserver has already parsing logic for our packets and would probably require the least amount of work to be formed into what we'd like to have. But of course go is a terrible language and not a lot of fun to write. If you have another favorite language you could also do this in that.

2) Imho 1) rules out having it as part of the game core or building on top of it - because this would mean doing it in c++ OR extending our build system to support another language. Both paths are unattractive IMHO.

Having it inside the metaserver is possible, but having it separate seems better to me: we could restart or even update the metaserver without games getting canceled and the game relays could crash also independently without affecting the metaserver.

Following these arguments I think a stand-alone binary (that maybe reuses code from the metaserver) is the most attractive solution. I do not foresee us bundling the relay with the game - I fully expect that Widelands will grow code for the relay which only makes sense on the server. Bundling it then will require another version of the relay anyways. But this seems tangential now: should bundling become useful we can always investigate this then.

3) I think we want something we fully control. I believe that we want to collect some kind of statistics like the number of games, mean length, mean ping time, memory consumption and CPU requirements on the server and so on into some kind of influxdb/grafana combination. For this we need to understand the traffic we relay. I also think that the relay can not be completely dumb for other reasons: for example properly handling reconnects when a player temporarily drops needs some sort of renegotiating that the relay has to do.

Gun, do you have opinions here?

[Notabilis (notabilis27)]

Thanks for the extensive answer! Admittedly I haven't considered security until now. Shame on me.
Regarding the language I guess we should select Go since the metaserver already is written in it. As the relay will be a separate program: Shall we create an own repository for it?

Currently I am thinking about one relay server process per running game. Is that also what you are planning or do we have one relay server handling all games? Having one per game would be easier to code and a crash would only break one game. The disadvantage is, that each relay would require its own (dynamic) port number.

[SirVer (sirver)]

> Regarding the language I guess we should select Go since the metaserver already is written in it.

Cool with me. I hate Go though :(

> As the relay will be a separate program: Shall we create an own repository for it?

naa, code reuse with the metaserver will be easier if it lives next to it. We can split it out when this causes us pain. I think semantically the binary will be tightly coupled to the metaserver anyways, so it might be a good place to live.

> Currently I am thinking about one relay server process per running game.

yup, that is how I envision it too.

> The disadvantage is, that each relay would require its own (dynamic) port number.

We'd have to poke holes into the firewall on the server for this to work - we'd probably want to open the iptables hole exactly for the relays lifetime and auto-close it on termination. No idea how this can be accomplished, this requires some research. Alternatively we could open a range of ports in the firewall permanently, but of course this is not very secure.

Another solution that only requires a single port would be to have a broker process that listen on the port, identifies the connection(s) somehow, spawns the relay and passes it the connection(s) through SCM_RIGHTS [1, 2].

Maybe something similar can also be achieved using SO_REUSEPORT, but I have no experience with that.

[1] https://stackoverflow.com/a/18938088/200945
[2] https://stackoverflow.com/a/4491203/200945

[GunChleoc (gunchleoc)]

I'm anything but a networking expert but running it separately and as sandboxed as possible sounds like a sensible plan to me.

It might be worth our time to check out how other games handle this, e.g. Megaglest, Battle for Wesnoth, 0AD.

0AD are currently working on the same problem: https://code.wildfiregames.com/D364

I don't know which part of the code is doing this for the other 2 projects, but I could ask on their forums.

[Notabilis (notabilis27)]

Having a broker sounds like an alternative, but I have no idea whether this is possible with Go. But it might be worth a try.
What is the problem with permanently open ports in the firewall? If no process is listening on them, it should not matter. And if someone manages to start an arbitrary process on the metaserver, we have pretty much lost anyway. Am I missing something?

Looking at other games is a good idea. You already mentioned 0AD some time ago in #6 and we (as in: I) decided against their approach (hole punching) since it is not reliable. Megaglest and OpenTTD require manual port forwardings.
Battle for Wesnoth seems to be using a relay server similar to our plans, according to this article:
http://www.aosabook.org/en/wesnoth.html
I haven't looked at the sources yet, might be interesting.

[GunChleoc (gunchleoc)]

I just talked to a friend who knows more about this kind of stuff. First of all, anything open is also hackable, so we need to be aware of that.

The recommended looking at utorrent who were the first to implement a peer-to-peer solution to this problem, where they establish a direct connection without the need of a server or manual forwarding of ports.

He also pointed out that IP addresses are personally identifiable information and that we might run into trouble with the data protection act if we collect information about IP addresses on the server end.

[SirVer (sirver)]

> What is the problem with permanently open ports in the firewall?

Not a problem per se, but it reduces security. A malicious tool (for example a botnet part) no longer needs to get root to open up a port in the firewall. It can run as any user and find a open port to the internet. So it reduces security a bit.

> And if someone manages to start an arbitrary process on the metaserver, we have pretty much lost anyway.

That is true, though there are still different qualities: having root or not for example. And generally security in depth is the philosophy of assuming breakage and still trying to secure the system somehow.

The wesnoth implementation is probably very interesting. We can learn how they deal with the port problem and if they use a relay or start one process per game. Their problem is much simpler though, since they are not real-time, but round based.

[Notabilis (notabilis27)]

uTorrent is using UPNP (automatic port forwardings), which is not enabled in all routers. So it is not enough when we want something that always works.
The pointer with the data protection act is a good one. When we start collecting statistics we should look into legal issues before we implement something. But I guess (not know!) that for the relay server as such it doesn't matters.

From looking at their sources, it seems that Wesnoth is using a single metaserver/relay instance for all games. So they have one process with only one open port for it.
I can't say whether a single relay server is possible for Widelands, too. Until we have it and try it out, I don't think we can say much about how busy the relay is with one game and how many games it might be able to manage. But it should be possible to design the protocol and the relay in a way that it can easily be upgraded to support multiple games at once, if we want to try this path. Of course, that would mean that a crash of the (single) relay process breaks all currently running games.

[Notabilis (notabilis27)]

I wrote up some documentation for a communication protocol between server/client and the relay. The protocol is really basic and only deals with simply relaying of packets for one game. Features like better reconnects or late joins are not supported. If no-one objects, I will try to implement client/server and relay for this protocol. When the basics work, we can decide on how we support multiple games.

[SirVer (sirver)]

Could you put the The file in a branch? That way we could comment directly in the source code. I think that would be easiest.

Thanks for working on this! I will not be able to look through the document till Monday though.

> Am 15.07.2017 um 18:34 schrieb Notabilis <email address hidden>:
>
> I wrote up some documentation for a communication protocol between
> server/client and the relay. The protocol is really basic and only deals
> with simply relaying of packets for one game. Features like better
> reconnects or late joins are not supported. If no-one objects, I will
> try to implement client/server and relay for this protocol. When the
> basics work, we can decide on how we support multiple games.
>
> ** Attachment added: "relay_protocol.h"
> https://bugs.launchpad.net/widelands/+bug/1689087/+attachment/4915246/+files/relay_protocol.h
>
> --
> You received this bug notification because you are subscribed to
> widelands.
> https://bugs.launchpad.net/bugs/1689087
>
> Title:
> Implementing a relay server
>
> Status in widelands:
> Confirmed
>
> Bug description:
> Two short notices before you continue reading:
> - This bug report is more like a project proposal/description but launchpad blueprints do not seem to have a way to discuss them.
> - The following plans are concerned with the underlying TCP connections of the network gaming and do not affect the gameplay as such.
>
> Currently, when hosting an internet game of widelands the host player has to create port forwarding rules in his router. For a lot of players this is a problem. As a possible solution the introduction of a relay server was proposed quite some time ago [1]. The basic idea is that all players (even the host) are connecting to this globally reachable relay so they don't require any port forwardings on their own systems.
> [1] https://blueprints.launchpad.net/widelands/+spec/metaserver-game-communication-module
>
> Another older request is support for IPv6 [2], which will probably become even more pressing in the future. Since this seems to require a change of the used networking library I would prefer to do it before starting to implement a relay server so we can use the same technology for both.
> [2] https://bugs.launchpad.net/widelands/+bug/1092567
>
> In the following I will describe what I am currently planning to do.
> If you spot any problems or have other comments, feel free to tell me.
>
>
> == Refactoring Step 1 ==
> I renamed NetHost/NetClient to GameHost/GameClient and am working on extracting the network specific parts into new NetHost/NetClient classes (no branch online yet). The plan is to have all network calls hidden so they can be easily replaced.
>
> == Using Boost.Asio ==
> When the classes are extracted, I intend to replace SDL_net with Boost.Asio since there is no IPv6 for SDL_net (only an unofficial modification for an outdated version of SDL_net).
>
> == IPv6 ==
> When we already use Boost.Asio, it should be quite easy to add support for IPv6 in the relevant wideland classes. I haven't tried it out yet but I think it should even be possible to support IPv4 *and* IPv6 clients in the same game without any speci...

[Notabilis (notabilis27)]

Okay, is uploaded as:
https://code.launchpad.net/~widelands-dev/widelands/net-relay/+merge/327491
Its not urgent, I just got some nice bugs to work on.

[SirVer (sirver)]

This is now implemented and merged. Thanks for your hard work and dedication, Notabilis!

Gun, do you want a new bug to track the necessary documentation update in the Wiki? No more port forwarding and ipv6 support forever!!!

[Notabilis (notabilis27)]

Great that this is online! Seems to work fine so far.

When we update the documentation we have to keep in mind, that the current "default", i.e., build 19, still requires port forwarding. So we need one complicated documentation for port forwarding in build 19 and a nice simple "it just works" documentation for trunk and build 20+.

[GunChleoc (gunchleoc)]

Thanks for this great feature :)

I have updated the documentation on the wiki with notes that port forwarding won't be needed after Build 19.

[GunChleoc (gunchleoc)]

Fixed in build20-rc1