Linux 5.4.0-144-generic x86_64 multithread / fork issues

[Alfredo Pardo]

We have same servers already running in 5.4.0-122 , 126, 128 and 131 releases with no issues whatsoever and exceptional good performance. They basically have either Apache/2.4.41/mod_php7.4 or Apache/2.4.41/mpm_event with phm7.4-fpm setups, all working perfect.

When we load kernel 5.4.0-144 CPU processing go thru the roof, and process kworker/u17:0-xprtiod show at the top. Both setups perform badly with Apache performance going avg 200ms/req in older kernels to over 4,500ms/req in this 144 release. CPU usage from avg 0.4% to over 50% under same traffic and requests rate.

We just reloaded old kernels back, via /etc/default/grub, and performance is right back. Same packages, same services, everything the same but the Linux kernel.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-144-generic 5.4.0-144.161
ProcVersionSignature: Ubuntu 5.4.0-144.161-generic 5.4.229
Uname: Linux 5.4.0-144-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 9 13:21 seq
 crw-rw---- 1 root audio 116, 33 Mar 9 13:21 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.25
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: pass
Date: Thu Mar 9 13:22:31 2023
InstallationDate: Installed on 2021-11-17 (477 days ago)
InstallationMedia: Ubuntu-Server 20.04.3 LTS "Focal Fossa" - Release amd64 (20210824)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
 Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Lsusb-t:
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
     |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
     |__ Port 2: Dev 3, If 0, Class=Hub, Driver=hub/7p, 12M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
MachineType: VMware, Inc. VMware Virtual Platform
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 svgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-144-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro maybe-ubiquity
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-144-generic N/A
 linux-backports-modules-5.4.0-144-generic N/A
 linux-firmware 1.187.36
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/12/2018
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd12/12/2018:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Matthew Ruffell (mruffell)]

Hi Alfredo,

Do you happen to use NFSv3 in your environment? There was a regression in 5.4.0-144-generic that cased a massive spike in ACCESS requests being made, which could explain low performance.

bug 2009325

Thanks,
Matthew

[Christian Fertig (fertig)]

we have the same issue with this kernel, but also with NFSv4

thanks,
Christian

[Alfredo Pardo (tsgesm)]

Hi Matthew,

yes we do have NFSv3 in our environment and mostly all PHP requests pull files sitting in network storage. However "waiting access" (wa) indicator is close to zero while inspecting "top".

In the past when we had had issues with NFS3 most processes slow down while waiting for I/O thus causing a significant spike in the indicator.

Doesn't happen this time.

Regards,
Al

[Matthew Ruffell (mruffell)]

Hi Christian, Alfredo,

5.4.0-146-generic was released yesterday with a fix for the NFS regression, and it should have less bandwidth consumption than 5.4.0-144-generic.

There is still some reports that it still has higher NFS requests than before the regression was introduced, so please chime in on the below bug if the situation improves with 5.4.0-146-generic, but is not completely addressed.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325

Thanks,
Matthew

[Alfredo Pardo (tsgesm)]

Thanks Matt. I can confirm our issue is completely gone and performance is back with 5.4.0-146-generic.

Alfredo