In short, we faced an issue with a huge IO wait on a bionic Ubuntu 4.15.0-118.119-generic kernel.
This is the full list of process and the kernel function they were stuck in [2].
The main issue can probably be summarized by this perf reports
* first identify that the cpu are stuck in idle because of something[1]
* second, see what kernel function seems to stuck the process kswapd0 and kswapd1 [2].
We could see that this seems to be the mutex_lock in the bch_mca_scan function [3].
Hello,
In short, we faced an issue with a huge IO wait on a bionic Ubuntu 4.15.0- 118.119- generic kernel.
This is the full list of process and the kernel function they were stuck in [2].
The main issue can probably be summarized by this perf reports
* first identify that the cpu are stuck in idle because of something[1]
* second, see what kernel function seems to stuck the process kswapd0 and kswapd1 [2].
We could see that this seems to be the mutex_lock in the bch_mca_scan function [3].
After running the command:
| sudo bash -c "echo 1 > /sys/fs/ bcache/ f1a1e8cb- 3e6b-40ea- 852e-583c48d0c2 b8/internal/ btree_shrinker_ disabled"
The server started to respond normally and the IO wait dropped significantly /pastebin. canonical. com/p/wYYKwHdRX k/ /pastebin. canonical. com/p/n2Tw57QyB C/ /pastebin. canonical. com/p/3QqFTfdHh X/ /git.launchpad. net/~ubuntu- kernel/ ubuntu/ +source/ linux/+ git/bionic/ tree/drivers/ md/bcache/ btree.c? h=Ubuntu- 4.15.0- 118.119# n674
[0]: https:/
[1]: https:/
[2]: https:/
[3]: https:/
======= ======= ====== signature 118.119- generic 4.15.18
$ cat /proc/version_
Ubuntu 4.15.0-
ProblemType: Bug 4.15.0- 118-generic 4.15.0-118.119 ature: User Name 4.15.0- 118.119- generic 4.15.18
DistroRelease: Ubuntu 18.04
Package: linux-image-
ProcVersionSign
Uname: Linux 4.15.0-118-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 29 10:04 seq
crw-rw---- 1 root audio 116, 33 Sep 29 10:04 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.16
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Tue Oct 6 20:36:18 2020
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: HP ProLiant DL380 G7
PciMultimedia:
ProcFB: 0 radeondrmfb /boot/vmlinuz- 4.15.0- 118-generic root=UUID= c6ad1629- a506-4043- a339-6d57f0708d 12 ro console= ttyS1,115200 nosplash ersions: restricted- modules- 4.15.0- 118-generic N/A backports- modules- 4.15.0- 118-generic N/A bvrP67: bd05/05/ 2011:svnHP: pnProLiantDL380 G7:pvr: cvnHP:ct23: cvr:
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.173.18
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to bionic on 2019-09-27 (375 days ago)
dmi.bios.date: 05/05/2011
dmi.bios.vendor: HP
dmi.bios.version: P67
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL380 G7
dmi.sys.vendor: HP