Wireguard is broken after yesterdays apt-get dist-upgrade

[Alexander Gaengel]

[Ubuntu server 18.04.1]
Yesterdays apt-get update && apt-get dist-upgrade broke wireguard on all systems. This is a little annoying if some of this systems are located 1600km away and the wireguard-vpn is used to reach them...

Packages relevant upgraded:
linux-image-5.0.0-37-generic -> linux-image-5.3.0-26-generic

The wireguard-dkms was rebuilded during this upgrade, and loads into the new kernel without problems:
[ 5.038245] wireguard: loading out-of-tree module taints kernel.
[ 5.038396] wireguard: module verification failed: signature and/or required key missing - tainting kernel
[ 5.039066] wireguard: WireGuard 0.0.20191219 loaded. See www.wireguard.com for information.
[ 5.039067] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <email address hidden>. All Rights Reserved.

Wireguard is configured through systemd-networkd, the config has not changed since months. There are no errors/warnings other signs of problems regarding wireguard in any log. But Wireguard just don't transfer any data, with TCP-Dump I can see that the wireguard-server don't answer to any of the packets that are arriving from not affected clients. And affected clients are not sending out any data to the wireguard server.

One thing is eye-catching, in the output of wg on the affected devices the peers are missing:
Affected device:
interface: wg0
  public key: xxx
  private key: (hidden)
  listening port: 443

Not affected device:
interface: wg0
  public key: xxx
  private key: (hidden)
  listening port: 443

peer: xxx
  preshared key: (hidden)
  endpoint: xxx
  allowed ips: xxx
  transfer: 0 B received, 2.89 KiB sent
  persistent keepalive: every 25 seconds

Config of one device:
30-wg0.netdev:
[NetDev]
Name = wg0
Kind = wireguard
Description = Wireguard

[WireGuard]
PrivateKey = xxx
# PublicKey = xxx
ListenPort = xxx

[WireGuardPeer]
PublicKey = xxx
PresharedKey = xxx
AllowedIPs = xxx
Endpoint = xxx
PersistentKeepalive = 25

30-wg0.network:
[Match]
Name = wg0

[Link]
MTUBytes=1300

[Network]
DNS = xxx
DNS = xxx
Domains=xxx

[Address]
Address = xxx

[Address]
Address = xxx

[Route]
Gateway = xxx
Destination = xxx
Metric=50000

Full List of Packages updated:
Start-Date: 2020-01-17 14:40:50
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef --force-yes dist-upgrade
Install: linux-image-5.3.0-26-generic:amd64 (5.3.0-26.28~18.04.1, automatic), linux-headers-5.3.0-26:amd64 (5.3.0-26.28~18.04.1, automatic), linux-headers-5.3.0-26-generic:amd64 (5.3.0-26.28~18.04.1, automatic), linux-modules-extra-5.3.0-26-generic:amd64 (5.3.0-26.28~18.04.1, automatic), linux-modules-5.3.0-26-generic:amd64 (5.3.0-26.28~18.04.1, automatic)
Upgrade: php7.2-bz2:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-common:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-cli:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-fpm:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-mysql:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), linux-headers-generic-hwe-18.04:amd64 (5.0.0.37.95, 5.3.0.26.95), php7.2-sqlite3:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-json:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-opcache:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-curl:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-xml:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-intl:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-zip:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-mbstring:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-readline:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), php7.2-gd:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), linux-image-generic-hwe-18.04:amd64 (5.0.0.37.95, 5.3.0.26.95), libdrm2:amd64 (2.4.97-1ubuntu1~18.04.1, 2.4.99-1ubuntu1~18.04.1), linux-generic-hwe-18.04:amd64 (5.0.0.37.95, 5.3.0.26.95), php7.2-pgsql:amd64 (7.2.24-0ubuntu0.18.04.1, 7.2.24-0ubuntu0.18.04.2), libdrm-common:amd64 (2.4.97-1ubuntu1~18.04.1, 2.4.99-1ubuntu1~18.04.1)
End-Date: 2020-01-17 14:46:41

[Alexander Gaengel (agauntu)]

Update: If I build a config with my data to according to the wg manpage and pass it to wg setconf, I See the peers again with wg. So, maybe it is a problem between systemd-networkd and the new 5.3 kernel?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Theodis Butler (djeraseit)]

FYI, I'm having the same issue on CentOS 8 as well.