Slow disk IO after upgrade to kernel 5.15.0-112-generic

[Jan Vojt]

I noticed performance issues on the server running some backend applications after updating OS packages with Ubuntu Jammy. After further investigation, the slowness was caused by disk IO.

After the update, running on kernel 5.15.0-112, the disk writes take about twice as long. I can reproduce this by changing the GRUB configuration to boot into kernel 5.15.0-107, where the writes are consistently faster than on the newer kernel. See below.

# uname -a
Linux dev03 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# time dd if=/dev/urandom of=deleteme bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB, 977 MiB) copied, 6.54711 s, 156 MB/s

real 0m6.782s
user 0m0.192s
sys 0m5.968s

# uname -a
Linux dev03 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
# time dd if=/dev/urandom of=deleteme bs=1024 count=1000000
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB, 977 MiB) copied, 3.77936 s, 271 MB/s

real 0m3.781s
user 0m0.132s
sys 0m3.648s

These times are with NVMe drives on ext4 filesystem. I observe the same performance difference for ZFS filesystem, and also with HDDs (spinning drives). The disk performance is also worse for 6.5.0-41 kernel. I also tried 6.5.0-35 and linux-image-generic-hwe-22.04 - same problem - writes are slow.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-5.15.0-112-generic 5.15.0-112.122
ProcVersionSignature: Ubuntu 5.15.0-112.122-generic 5.15.152
Uname: Linux 5.15.0-112-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices:
 total 0
 drwxr-xr-x 2 root root 60 Jun 24 08:23 by-path
 crw-rw---- 1 root audio 116, 2 Jun 24 08:23 controlC0
 crw-rw---- 1 root audio 116, 1 Jun 24 08:23 seq
 crw-rw---- 1 root audio 116, 33 Jun 24 08:23 timer
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.15.0-112-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: unknown
Date: Mon Jun 24 10:24:44 2024
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
 /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
 /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-112-generic root=UUID=3bff164a-d728-43bd-a926-ad3be6ff4715 ro consoleblank=0 systemd.show_status=true nomodeset consoleblank=0
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-112-generic N/A
 linux-backports-modules-5.15.0-112-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.31
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/09/2020
dmi.bios.release: 5.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: L1.02W
dmi.board.name: B450D4U-V1L
dmi.board.vendor: ASRockRack
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrL1.02W:bd07/09/2020:br5.14:svnHetzner:pn:pvr1.0:rvnASRockRack:rnB450D4U-V1L:rvr:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:sku:
dmi.product.version: 1.0
dmi.sys.vendor: Hetzner

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jan Vojt (jan-m)]

I did git bisect to identify which change in the code repo caused the performance degradation. The log is below. It identifies 3 commits, which contain a small change around RETHUNK feature. The performance degradation might therefore be intentional to avoid security vulnerabilities.

# git bisect log
git bisect start
# bad: [c6f48204c530ee031a62a37cf72dc91bdd3e0c3e] UBUNTU: Ubuntu-5.15.0-111.121
git bisect bad c6f48204c530ee031a62a37cf72dc91bdd3e0c3e
# good: [7d5ccdf5683ced7ab2a337434a23996029338946] UBUNTU: Ubuntu-5.15.0-107.117
git bisect good 7d5ccdf5683ced7ab2a337434a23996029338946
# good: [ef10ffdd4e8bebdc7ae915a897fa6aa11668882c] UBUNTU: Ubuntu-5.15.0-106.116
git bisect good ef10ffdd4e8bebdc7ae915a897fa6aa11668882c
# bad: [69cb15bd0578150f9edaf8c26d004f08749c0d87] netrom: Fix a data-race around sysctl_netrom_obsolescence_count_initialiser
git bisect bad 69cb15bd0578150f9edaf8c26d004f08749c0d87
# bad: [0339b5bfd90480afce66a4e058c8c14d521be801] net: ethernet: ti: add missing of_node_put before return
git bisect bad 0339b5bfd90480afce66a4e058c8c14d521be801
# good: [a22959781a89cb5c10d9ea0a4a5f206fc47b363e] fs/ntfs3: Disable ATTR_LIST_ENTRY size check
git bisect good a22959781a89cb5c10d9ea0a4a5f206fc47b363e
# bad: [feb2673ad44c77ceda33f4573aba0a02f08cfaa6] usb: roles: don't get/set_role() when usb_role_switch is unregistered
git bisect bad feb2673ad44c77ceda33f4573aba0a02f08cfaa6
# good: [3ad4cf481b5f1e4ab9eb5c4fb1678b42d23e2046] platform/x86: intel-vbtn: Stop calling "VBDL" from notify_handler
git bisect good 3ad4cf481b5f1e4ab9eb5c4fb1678b42d23e2046
# good: [4bdbf7be6a24066bca519cfb2f8aeb1ec56c2e4e] x86/ftrace: Use alternative RET encoding
git bisect good 4bdbf7be6a24066bca519cfb2f8aeb1ec56c2e4e
# bad: [ee64cc371623f4eac6bf543b842a8956b419b9b6] usb: cdnsp: blocked some cdns3 specific code
git bisect bad ee64cc371623f4eac6bf543b842a8956b419b9b6
# bad: [fbdd7ca1b50202aa535202c30ea2bc5be50bd3e0] x86/alternative: Make custom return thunk unconditional
git bisect bad fbdd7ca1b50202aa535202c30ea2bc5be50bd3e0
# skip: [09ae122959920bbe0b441e49c24ffe75cf1a9637] Revert "x86/alternative: Make custom return thunk unconditional"
git bisect skip 09ae122959920bbe0b441e49c24ffe75cf1a9637
# skip: [68fa28ce22a624dfcd81fdd4813cd5f81f6f0f5a] x86/returnthunk: Allow different return thunks
git bisect skip 68fa28ce22a624dfcd81fdd4813cd5f81f6f0f5a
# only skipped commits left to test
# possible first bad commit: [fbdd7ca1b50202aa535202c30ea2bc5be50bd3e0] x86/alternative: Make custom return thunk unconditional
# possible first bad commit: [09ae122959920bbe0b441e49c24ffe75cf1a9637] Revert "x86/alternative: Make custom return thunk unconditional"
# possible first bad commit: [68fa28ce22a624dfcd81fdd4813cd5f81f6f0f5a] x86/returnthunk: Allow different return thunks