23 wireguard peers hang systemd-networkd

[Leonid Evdokimov]

I'm running Ubuntu 18.04.1 LTS with systemd=237-3ubuntu10.9.
Linux kernel version is 4.15.0-32-generic #35-Ubuntu SMP.
wireguard=0.0.20181218-wg1~bionic from PPA.

I have a Wireguard-based VPN server that has several peers. As soon as number of peers is 22 or lower, everything works okay. As soon as I add the 23rd peer, restart of `systemd-networkd` service fails with timeout while systemd-networkd hogs CPU.

Moreover, if I reboot the box while wireguard configuration is "broken", systemd-networkd fails to apply network settings on boot and the box is no longer accessible over the network.

Configuration is structured in a following way (keys are fake):

==> wg0.netdev
[NetDev]
Name=wg0
Kind=wireguard
Description=Wireguard VPN server
[WireGuard]
ListenPort=4500
PrivateKey=kNl7tkhCM1Crj8RhUIn8xvwcg+UoOkw26kQjQEtZk1k=
[WireGuardPeer]
PublicKey=AfM1AN4IIUe5AVypFg2pcNrQmqOtZQIJLgusbkDYXkI=
AllowedIPs=fd6f:b446:a2ca:0400:cb6f:b446:a2ca:bd0b/128
AllowedIPs=fd6f:b446:a2ca:cb6f:b446:a2ca::/96
# and 22 more [WireGuardPeer] like that

==> wg0.network
Name=wg0
[Network]
Address=fd6f:b446:a2ca:0400::1/64
[Route]
Destination=fd6f:b446:a2ca:cb6f:b446:a2ca::/96
# and 22 more [Route] sections like that

syslog logs are attached both for "good" and "bad" cases, sample of strace logs is also attached for "bad" case.

I'm filling the issue here as the aforementioned systemd version is already out of scope of upstream bug tracker per https://github.com/systemd/systemd/blob/master/docs/CONTRIBUTING.md#filing-issues

[Leonid Evdokimov (darkk)]

I've written a small script to reproduce the issue and I got following results using VMs from Linode:

- broken with systemd=237-3ubuntu10.9 on Ubuntu 18.04, 4.15.0-43-generic #46-Ubuntu.
- broken with systemd=237-3ubuntu10.11 on Ubuntu 18.04, 4.15.0-43-generic #46-Ubuntu.
- broken with systemd=239-7ubuntu10.6 on Ubuntu 18.10, 4.18.0-13-generic #14-Ubuntu.

The way I used to reproduce the issue was to 1) start Linode VM, 2) run `./systemd-wg 23`.

For some reason, 23 is the magic number of peers: 22 peers, 24 peers and 1500 peers work okay, but 23 peers trigger the bug.

Given the fact that 239 is also affected, I'm submitting it upstream.

[Joshua Sjoding (joshua.sjoding)]

As near as I can tell the fix for this was never backported from systemd v241 to bionic. I recently filed a related a bug report here:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1853956

My symptoms are a little different (misconfiguration instead of an infinite loop), but I have a strong suspicion that the underlying cause is the same.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue