I am seeing an issue with NFSv4 performance on Ubuntu 20.04.3 LTS on both clients and server (tested kernels 5.4 to 5.16), where the server is connected by 10Gbit Ethernet and with multiple clients connected by 1Gbit. I am reading a large file from an NFS mount via "dd" sending output to /dev/null with bs=4096. Using default sysctl and mount options I am seeing speeds maxing out below 1Gbit/sec. If I force NFSv3 I see speeds close to 10Gbit/sec with sufficient clients connected. I also see no issue with Ubuntu 16.04 (when used for both server and clients) in conjunction with NFSv4. I have attached the output from two iftop's which shows the status when using NFSv4 and when using NFSv3, in the NFSv4 you can clearly see one client reading at max speed and all the others apparently throttling back to practically nothing.
I have additionally tested a range of mount options, which made no difference, BBR congestion control which made no difference and the following kernel settings which also made no difference:
net.core.netdev_max_backlog=250000
net.core.rmem_max=4194304
net.core.wmem_max=4194304
net.core.rmem_default=4194304
net.core.wmem_default=4194304
net.core.optmem_max=4194304
net.ipv4.tcp_rmem=4096 87380 4194304
net.ipv4.tcp_wmem=4096 65536 4194304
net.ipv4.tcp_mem=786432 1048576 26777216
net.ipv4.udp_mem=1529892 2039859 3059784
net.ipv4.udp_rmem_min=16384
net.ipv4.udp_wmem_min=16384
The problem is seen on dissimilar hardware, ie this problem exists when testing with an HP DL380 G10 with Mellanox 10Gbit Ethernet connected to Cisco switch, and also on a Dell R430 with Broadcom 10Gbit Ethernet connected to a Netgear switch (just to name two of several configurations that have been tested). The clients vary in each test case also, but are desktop PCs and laptops.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw----+ 1 root audio 116, 1 Feb 15 10:23 seq
crw-rw----+ 1 root audio 116, 33 Feb 15 10:23 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.21
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2022-02-14 (0 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Dell Inc. PowerEdge R430
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=d227a745-134e-4633-9c26-bcc062655f95 ro
ProcVersionSignature: Ubuntu 5.13.0-28.31~20.04.1-generic 5.13.19
RelatedPackageVersions:
linux-restricted-modules-5.13.0-28-generic N/A
linux-backports-modules-5.13.0-28-generic N/A
linux-firmware 1.187.26
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: focal uec-images
Uname: Linux 5.13.0-28-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 06/07/2021
dmi.bios.release: 2.13
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.13.0
dmi.board.name: 0CN7X8
dmi.board.vendor: Dell Inc.
dmi.board.version: A04
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.13.0:bd06/07/2021:br2.13:svnDellInc.:pnPowerEdgeR430:pvr:rvnDellInc.:rn0CN7X8:rvrA04:cvnDellInc.:ct23:cvr:skuSKU=NotProvided;ModelName=PowerEdgeR430:
dmi.product.name: PowerEdge R430
dmi.product.sku: SKU=NotProvided;ModelName=PowerEdge R430
dmi.sys.vendor: Dell Inc.
Where both client and server are connected via 10Gbit speeds are as expected with NFSv4 (testing with a single client)