Looks like the same issue kcompactd and btrfs-transaction locking up. Can not immediately reproduce, seemingly takes ~2weeks to appear. More problematic systems generally also have higher disk i/o, all btrfs.
%Cpu0 : 3.8 us, 1.3 sy, 0.0 ni, 0.0 id, 94.9 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 1.3 us, 2.6 sy, 0.0 ni, 96.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%CPU PID TID TIME STAT RSS SZ F COMMAND 0.0 27 27 00:00:03 D 0 0 1 [kcompactd0] 0.0 618 618 00:03:04 D 0 0 1 [btrfs-transacti]
cat /proc/27/stack [<0>] __lock_page+0xff/0x140 [<0>] migrate_pages+0x91f/0xb80 [<0>] compact_zone+0x681/0x950 [<0>] kcompactd_do_work+0xfe/0x2a0 [<0>] kcompactd+0x86/0x1c0 [<0>] kthread+0x121/0x140 [<0>] ret_from_fork+0x35/0x40 [<0>] 0xffffffffffffffff
cat /proc/618/stack [<0>] lock_extent_buffer_for_io+0x100/0x2a0 [btrfs] [<0>] btree_write_cache_pages+0x1b8/0x420 [btrfs] [<0>] btree_writepages+0x5d/0x70 [btrfs] [<0>] do_writepages+0x4b/0xe0 [<0>] __filemap_fdatawrite_range+0xcf/0x100 [<0>] filemap_fdatawrite_range+0x13/0x20 [<0>] btrfs_write_marked_extents+0x68/0x140 [btrfs] [<0>] btrfs_write_and_wait_marked_extents.constprop.20+0x4f/0x90 [btrfs] [<0>] btrfs_commit_transaction+0x696/0x910 [btrfs] [<0>] transaction_kthread+0x18d/0x1b0 [btrfs] [<0>] kthread+0x121/0x140 [<0>] ret_from_fork+0x35/0x40 [<0>] 0xffffffffffffffff
INFO: task kcompactd0:27 blocked for more than 120 seconds. Tainted: P E 4.15.0-122-generic #124-Ubuntu "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kcompactd0 D 0 27 2 0x90000000 Call Trace: __schedule+0x24e/0x880 schedule+0x2c/0x80 io_schedule+0x16/0x40 __lock_page+0xff/0x140 ? page_cache_tree_insert+0xe0/0xe0 migrate_pages+0x91f/0xb80 ? __ClearPageMovable+0x10/0x10 ? isolate_freepages_block+0x3b0/0x3b0 compact_zone+0x681/0x950 kcompactd_do_work+0xfe/0x2a0 ? __switch_to_asm+0x35/0x70 ? __switch_to_asm+0x41/0x70 kcompactd+0x86/0x1c0 ? kcompactd+0x86/0x1c0 ? wait_woken+0x80/0x80 kthread+0x121/0x140 ? kcompactd_do_work+0x2a0/0x2a0 ? kthread_create_worker_on_cpu+0x70/0x70 ret_from_fork+0x35/0x40 ~
Looks like the same issue kcompactd and btrfs-transaction locking up. Can not immediately reproduce, seemingly takes ~2weeks to appear. More problematic systems generally also have higher disk i/o, all btrfs.
%Cpu0 : 3.8 us, 1.3 sy, 0.0 ni, 0.0 id, 94.9 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 1.3 us, 2.6 sy, 0.0 ni, 96.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%CPU PID TID TIME STAT RSS SZ F COMMAND
0.0 27 27 00:00:03 D 0 0 1 [kcompactd0]
0.0 618 618 00:03:04 D 0 0 1 [btrfs-transacti]
cat /proc/27/stack page+0xff/ 0x140 pages+0x91f/ 0xb80 zone+0x681/ 0x950 do_work+ 0xfe/0x2a0 0x86/0x1c0 fork+0x35/ 0x40
[<0>] __lock_
[<0>] migrate_
[<0>] compact_
[<0>] kcompactd_
[<0>] kcompactd+
[<0>] kthread+0x121/0x140
[<0>] ret_from_
[<0>] 0xffffffffffffffff
cat /proc/618/stack buffer_ for_io+ 0x100/0x2a0 [btrfs] cache_pages+ 0x1b8/0x420 [btrfs] s+0x5d/ 0x70 [btrfs] 0x4b/0xe0 fdatawrite_ range+0xcf/ 0x100 fdatawrite_ range+0x13/ 0x20 marked_ extents+ 0x68/0x140 [btrfs] and_wait_ marked_ extents. constprop. 20+0x4f/ 0x90 [btrfs] transaction+ 0x696/0x910 [btrfs] kthread+ 0x18d/0x1b0 [btrfs] fork+0x35/ 0x40
[<0>] lock_extent_
[<0>] btree_write_
[<0>] btree_writepage
[<0>] do_writepages+
[<0>] __filemap_
[<0>] filemap_
[<0>] btrfs_write_
[<0>] btrfs_write_
[<0>] btrfs_commit_
[<0>] transaction_
[<0>] kthread+0x121/0x140
[<0>] ret_from_
[<0>] 0xffffffffffffffff
INFO: task kcompactd0:27 blocked for more than 120 seconds. kernel/ hung_task_ timeout_ secs" disables this message. 0x24e/0x880 0x16/0x40 page+0xff/ 0x140 tree_insert+ 0xe0/0xe0 pages+0x91f/ 0xb80 ble+0x10/ 0x10 freepages_ block+0x3b0/ 0x3b0 zone+0x681/ 0x950 do_work+ 0xfe/0x2a0 to_asm+ 0x35/0x70 to_asm+ 0x41/0x70 0x86/0x1c0 0x86/0x1c0 0x80/0x80 0x121/0x140 do_work+ 0x2a0/0x2a0 create_ worker_ on_cpu+ 0x70/0x70 fork+0x35/ 0x40
Tainted: P E 4.15.0-122-generic #124-Ubuntu
"echo 0 > /proc/sys/
kcompactd0 D 0 27 2 0x90000000
Call Trace:
__schedule+
schedule+0x2c/0x80
io_schedule+
__lock_
? page_cache_
migrate_
? __ClearPageMova
? isolate_
compact_
kcompactd_
? __switch_
? __switch_
kcompactd+
? kcompactd+
? wait_woken+
kthread+
? kcompactd_
? kthread_
ret_from_
~