Massive (3x) performance regression for WireGuard in Ubuntu 20.04 and 22.04

[James Wynn]

I have fully reproducible steps to demonstrate this issue on a vanilla DigitalOcean droplet, minimal WireGuard configuration and no firewall rules. I've also seen this issue on other hosting providers.

Testing with `iperf3 -c XXX -P 5`:

- Unencrypted traffic on DigitalOcean's VPC = ~2Gbps
- WireGuard Ubuntu 18.04 = ~1.3Gbps
- WireGuard Ubuntu 20.04 = ~400Mbps
- WireGuard Ubuntu 22.04 = ~400Mbps

htop reported only 20-30% load on the vCPU core so it isn't CPU-bound.

After doing these tests, I did them all again on a different day to rule out temporary network congestion.

Steps to reproduce below. Repeat with each Ubuntu version.

0. Create a DigitalOcean account.
1. Create two $6 droplets (eg, LON1 region) with Regular CPU & 1GB RAM each, called test01 & test02.
2. `apt-get update && DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y && reboot`
3. `apt-get install -y wireguard iperf3`

4. On test01, create `/etc/wireguard/test.conf` with these contents. Replace `YYY` with the IP address of the eth1 interface (VPC) on test02.

--------------------
[Interface]
PrivateKey = wOEa8/RS2v065wgYGQn5k7FqOXuZJ9aC/6NDW569c3g=
Address = 192.168.200.10/24
ListenPort = 51820
SaveConfig = false

[Peer]
PublicKey = wdXOzBptLD/QMZjhG475GErrz95Vpj4S7JPEwzcDMV8=
PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
Endpoint = YYY:51820
AllowedIPs = 192.168.200.20/32
--------------------

5. On test02, create `/etc/wireguard/test.conf` with these contents. Replace `XXX` with the IP address of the eth1 interface (VPC) on test01.

--------------------
[Interface]
PrivateKey = kCJ/4rVDTy86HxP9N5wUmgMF1Esqjc051jQPGhrQIGw=
Address = 192.168.200.20/24
ListenPort = 51820
SaveConfig = false

[Peer]
PublicKey = s/GtXkHOtPsqcNDy0BSRoMuxXYb4hK18dsQdkZk20yQ=
PresharedKey = j5Oeyhu/qDag2LunpVlFqKycp/9CH+Izjza5aq2cYss=
Endpoint = XXX:51820
AllowedIPs = 192.168.200.10/32
--------------------

6. On both droplets, run `systemctl start wg-quick@test`
7. On test01, run `iperf3 -s -B XXX`.
8. On test02, run `iperf3 -c XXX -P 5 -t 30` and observe ~2Gbps.
9. On test01, run `iperf3 -s -B 192.168.200.10`
10. On test02, run `iperf3 -c 192.168.200.10 -P 5 -t 30`

In steps 7 and 8, replace XXX with the IP address of the eth1 interface on test01.

[Andreas Hasenack (ahasenack)]

Are these droplets running an ubuntu kernel, or a Digital Ocean kernel? The type of interface (virtio) also matters.

[James Wynn (jameswynn)]

@ahasenack

The kernel is the ubuntu kernel:

# uname -a
Linux test01 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

DigitalOcean uses KVM:

# systemd-detect-virt
kvm

I'm not sure how to answer your virtio question. Maybe this helps:

# lsmod | grep virtio
virtio_net 61440 0
net_failover 20480 1 virtio_net
virtio_blk 20480 2
virtio_scsi 24576 0

[James Wynn (jameswynn)]

Oh, I found how to see if it's a virtio network interface. Both eth0 (public IP) and eth1 (VPC) are virtio interfaces.

# lshw -C net
  *-network:0
       description: Ethernet controller
       product: Virtio network device
       vendor: Red Hat, Inc.
       physical id: 3
       bus info: pci@0000:00:03.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix bus_master cap_list
       configuration: driver=virtio-pci latency=0
       resources: irq:10 ioport:c1c0(size=32) memory:fd012000-fd012fff memory:febe8000-febebfff
     *-virtio0
          description: Ethernet interface
          physical id: 0
          bus info: virtio@0
          logical name: eth0
          serial: de:bc:2c:65:xx:xx
          capabilities: ethernet physical
          configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 ip=x.x.x.x link=yes multicast=yes

  *-network:1
       description: Ethernet controller
       product: Virtio network device
       vendor: Red Hat, Inc.
       physical id: 4
       bus info: pci@0000:00:04.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: msix bus_master cap_list
       configuration: driver=virtio-pci latency=0
       resources: irq:11 ioport:c1e0(size=32) memory:fd013000-fd013fff memory:febec000-febeffff
     *-virtio1
          description: Ethernet interface
          physical id: 0
          bus info: virtio@1
          logical name: eth1
          serial: 2a:5c:f8:73:xx:xx
          capabilities: ethernet physical
          configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 ip=10.x.x.x link=yes multicast=yes
--------------------

Here's some other information if it helps.

# lscpu
Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Address sizes: 40 bits physical, 48 bits virtual
  Byte Order: Little Endian
CPU(s): 1
  On-line CPU(s) list: 0
Vendor ID: GenuineIntel
  Model name: DO-Regular
    CPU family: 6
    Model: 63
    Thread(s) per core: 1
    Core(s) per socket: 1
    Socket(s): 1
    Stepping: 2
    BogoMIPS: 4589.21
    Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_
                         2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invp
                         cid xsaveopt arat md_clear
Virtualization features:
  Virtualization: VT-x
  Hypervisor vendor: KVM
  Virtualization type: full
Caches (sum of all):
  L1d: 32 KiB (1 instance)
  L1i: 32 KiB (1 instance)
  L2: 4 MiB (1 instance)
NUMA:
  NUMA node(s): 1
  NUMA node0 CPU(s): 0
Vulnerabilities:
  Itlb multihit: KVM...

[Andreas Hasenack (ahasenack)]

In the Ubuntu 20.04 and 22.04 cases, is the unencrypted traffic on the VPC also still around 2Gbps?

[James Wynn (jameswynn)]

@ahasenack Yes, on all three Ubuntu versions (18.04, 20.04, 22.04) with the latest Ubuntu kernel for the respective Ubuntu release (as of today), the VPC traffic unencrypted hovers around 2Gbps.

[Seth Forshee (sforshee)]

I did some testing in DigitalOcean's nyc3 region. I did see a difference in performance between 18.04 and 20.04, though not as large as the reporter, with 20.04 being 30-50% slower than 18.04. Experimenting with different combinations I found that if the iperf3 receiver is on a host running the 5.15.0-69-generic kernel I see lower throughput, regardless of whether the sender is running 5.15.0-69-generic or 4.15.0-208-generic. This also holds true if I install 5.15.0-69-generic in 18.04. So it seems the regression is in receive throughput.