workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND

[Robert Ross]

dmesg has multiple messages:
[ 852.580542] evict_inodes inode 00000000be24a21a, i_count = 1, was skipped!
[ 852.580543] evict_inodes inode 000000002aea522e, i_count = 1, was skipped!
[ 852.580544] evict_inodes inode 0000000048d75a5c, i_count = 1, was skipped!
[ 852.580545] evict_inodes inode 00000000d8a9e4c2, i_count = 1, was skipped!
[ 852.580546] evict_inodes inode 000000003d2c905c, i_count = 1, was skipped!
[ 852.580547] evict_inodes inode 00000000e5b1b232, i_count = 1, was skipped!
[ 852.580548] evict_inodes inode 0000000097383a6b, i_count = 1, was skipped!
[ 852.580549] evict_inodes inode 00000000ca8e2b44, i_count = 1, was skipped!
[ 1751.869281] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[ 1781.467278] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 1806.065364] workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND
[ 1901.993975] evict_inodes inode 00000000abaa740e, i_count = 1, was skipped!
[ 1901.993981] evict_inodes inode 00000000515b9bf8, i_count = 1, was skipped!
[ 1901.993983] evict_inodes inode 000000001a69d536, i_count = 1, was skipped!
[ 1901.993984] evict_inodes inode 000000001403f675, i_count = 1, was skipped!
[ 1901.993985] evict_inodes inode 00000000757de21a, i_count = 1, was skipped!
[ 1901.993986] evict_inodes inode 000000000cee9028, i_count = 1, was skipped!
[ 1901.993987] evict_inodes inode 00000000d2827e77, i_count = 1, was skipped!

Description: Ubuntu 22.04.3 LTS
Release: 22.04

linux-oem-22.04d:
  Installed: 6.5.0.1004.4
  Candidate: 6.5.0.1004.4

Seems related to #2037214

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nazar Mokrynskyi (nazar-pc)]

Same is happening to me, probably related to BTRFS?

[Tonal (tonal-promsoft)]

see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2037214

[Gordon Lack (gordon-lack)]

The duplicate status is *clearly wrong*, as #2037214 refers to the "evict_inodes" issue (now fixed at some point) whereas this bug report is, according to its title, about the "workqueue: inode_switch_wbs_work_fn hogged CPU for >10000us 4" issue.

This has not been fixed (I'm seeing it on 6.8.0-31-generic in Noble).

There are solutions posted for it.
See:
 https://lkml.org/lkml/2023/8/30/613
 https://gitlab.freedesktop.org/drm/intel/-/issues/9245
 https://<email address hidden>/T/

which I picked up via: https://ubuntuforums.org/showthread.php?t=2490640

[Davor Grubisa (horzadome)]

Having this issue too.
For me the easiest way to reproduce it is to trigger btrfs balance and wait a minute or two, no other workloads running.

root@build01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble

Fully updated, running the latest available kernel
Linux build01 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

FWIW I haven't noticed this issue on any arch kernels running on similar setup and platform (btrfs data single, meta mirror, on nvmes amd 5950x).

[Matthias Dieter Wallnöfer (mwallnoefer)]

In my case I face the same issue with the "delayed_fput" module. And yes, I am also running btrfs (OS Ubuntu 22.04.4 LTS / Kernel 6.5.0-45-generic).

[ 208.207228] workqueue: delayed_fput hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[ 259.794567] workqueue: delayed_fput hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[ 378.217600] workqueue: delayed_fput hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND
[ 485.636678] workqueue: delayed_fput hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND

[maclenin (maclenin)]

I am also facing a similar "delayed_fput" issue. However, I am *not* running btrfs. (Xubuntu 22.04 / ext4 / 6.5.0-45-generic / Intel i5-9600K 3.7GHZ / 32gb ram).

journalctl output is to demonstrate an earlier pattern:

Apr 26 10:44:56 server kernel: workqueue: delayed_fput hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
Apr 26 11:18:25 server kernel: workqueue: delayed_fput hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
Apr 26 16:53:22 server kernel: workqueue: delayed_fput hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND
Apr 27 15:12:28 server kernel: workqueue: delayed_fput hogged CPU for >10000us 32 times, consider switching to WQ_UNBOUND
Apr 27 19:25:50 server kernel: workqueue: delayed_fput hogged CPU for >10000us 64 times, consider switching to WQ_UNBOUND
Apr 29 12:14:07 server kernel: workqueue: delayed_fput hogged CPU for >10000us 128 times, consider switching to WQ_UNBOUND
Apr 30 09:46:43 server kernel: workqueue: delayed_fput hogged CPU for >10000us 256 times, consider switching to WQ_UNBOUND
May 01 13:29:35 server kernel: workqueue: delayed_fput hogged CPU for >10000us 512 times, consider switching to WQ_UNBOUND

However, the issue persists:

Jul 29 15:10:55 server kernel: workqueue: delayed_fput hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
Jul 29 15:13:26 server kernel: workqueue: delayed_fput hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND

Could be workload-related: Running Chrome alone did not trigger, adding Chromium eventually triggers (~4 hours later).

Have experienced 2 recent (22 July / 25 July) crashes requiring hard re-start. Not certain whether related (to the "hog").