I have the same issue: kcompactd0 regularity (about every 10 days) blocks (iowait?) and never recovers. After a few more hours, the system becomes more or less dead. I never had this issue until, I would say, 2 month ago. I was running Ubuntu 18.04 fine until then. No heavy load on the server. # ps aux|grep compac root 28 0.0 0.0 0 0 ? D Aug27 0:06 [kcompactd0] btrfs-transaction seems OK though: # ps aux|grep trans root 471 0.0 0.0 0 0 ? S Aug27 7:01 [btrfs-transacti] From syslog: Sep 8 16:36:04 nc4 kernel: [1037582.945988] INFO: task kcompactd0:28 blocked for more than 120 seconds. Sep 8 16:36:04 nc4 kernel: [1037582.946216] Not tainted 4.15.0-112-generic #113-Ubuntu Sep 8 16:36:04 nc4 kernel: [1037582.946375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 8 16:36:04 nc4 kernel: [1037582.946557] kcompactd0 D 0 28 2 0x80000000 Sep 8 16:36:04 nc4 kernel: [1037582.946562] Call Trace: Sep 8 16:36:04 nc4 kernel: [1037582.946642] __schedule+0x24e/0x880 Sep 8 16:36:04 nc4 kernel: [1037582.946826] ? btree_releasepage+0x42/0x50 [btrfs] Sep 8 16:36:04 nc4 kernel: [1037582.946831] schedule+0x2c/0x80 Sep 8 16:36:04 nc4 kernel: [1037582.946833] io_schedule+0x16/0x40 Sep 8 16:36:04 nc4 kernel: [1037582.946853] __lock_page+0xff/0x140 Sep 8 16:36:04 nc4 kernel: [1037582.946861] ? page_cache_tree_insert+0xe0/0xe0 Sep 8 16:36:04 nc4 kernel: [1037582.946869] migrate_pages+0x91f/0xb80 Sep 8 16:36:04 nc4 kernel: [1037582.946875] ? __ClearPageMovable+0x10/0x10 Sep 8 16:36:04 nc4 kernel: [1037582.946881] ? isolate_freepages_block+0x3b0/0x3b0 Sep 8 16:36:04 nc4 kernel: [1037582.946884] compact_zone+0x681/0x950 Sep 8 16:36:04 nc4 kernel: [1037582.946887] kcompactd_do_work+0xfe/0x2a0 Sep 8 16:36:04 nc4 kernel: [1037582.946890] ? __switch_to_asm+0x35/0x70 Sep 8 16:36:04 nc4 kernel: [1037582.946892] ? __switch_to_asm+0x41/0x70 Sep 8 16:36:04 nc4 kernel: [1037582.946895] kcompactd+0x86/0x1c0 Sep 8 16:36:04 nc4 kernel: [1037582.946897] ? kcompactd+0x86/0x1c0 Sep 8 16:36:04 nc4 kernel: [1037582.946912] ? wait_woken+0x80/0x80 Sep 8 16:36:04 nc4 kernel: [1037582.946922] kthread+0x121/0x140 Sep 8 16:36:04 nc4 kernel: [1037582.946924] ? kcompactd_do_work+0x2a0/0x2a0 Sep 8 16:36:04 nc4 kernel: [1037582.946926] ? kthread_create_worker_on_cpu+0x70/0x70 Sep 8 16:36:04 nc4 kernel: [1037582.946929] ret_from_fork+0x35/0x40 Some more infos (all while kcompactd is still blocked): # uptime 17:20:25 up 12 days, 57 min, 1 user, load average: 1.48, 1.19, 1.08 # free -h total used free shared buff/cache available Mem: 5.8G 476M 1.6G 72M 3.8G 5.0G Swap: 6.0G 568M 5.4G # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.5 LTS Release: 18.04 Codename: bionic The root FS is BTRF on a LUKS encrypted volume. # cryptsetup status sda5_crypt /dev/mapper/sda5_crypt is active and is in use. type: LUKS1 cipher: aes-xts-plain64 keysize: 512 bits key location: dm-crypt device: /dev/sda5 sector size: 512 offset: 4096 sectors size: 1309712384 sectors mode: read/write flags: discards # pvs PV VG Fmt Attr PSize PFree /dev/mapper/sda5_crypt nc4-vg lvm2 a-- <624.52g 0 # vgs VG #PV #LV #SN Attr VSize VFree nc4-vg 1 2 0 wz--n- <624.52g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root nc4-vg -wi-ao---- <618.52g swap_1 nc4-vg -wi-ao---- 6.00g I can still write to disk: # time dd if=/dev/zero of=io-test bs=16k count=64k conv=fdatasync 65536+0 records in 65536+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.79033 s, 385 MB/s real 0m2.792s user 0m0.071s sys 0m1.858s # sar 11:05:01 AM CPU %user %nice %system %iowait %steal %idle 11:15:01 AM all 1.01 0.00 0.51 0.08 0.02 98.39 11:25:01 AM all 1.22 0.00 0.49 0.08 0.02 98.19 11:35:01 AM all 1.01 0.00 0.46 0.06 0.02 98.45 11:45:01 AM all 1.01 0.00 0.49 0.07 0.02 98.41 11:55:01 AM all 0.99 0.00 0.49 0.10 0.02 98.41 12:05:01 PM all 1.30 0.00 0.52 0.09 0.02 98.07 12:15:01 PM all 1.01 0.00 0.52 0.05 0.02 98.41 12:25:01 PM all 0.97 0.00 0.49 0.04 0.02 98.48 12:35:01 PM all 1.23 0.00 0.52 0.10 0.02 98.14 12:45:01 PM all 0.96 0.00 0.50 0.09 0.02 98.43 12:55:01 PM all 0.99 0.00 0.46 0.10 0.02 98.43 01:05:01 PM all 1.23 0.00 0.51 0.15 0.03 98.09 01:15:01 PM all 0.97 0.00 0.46 0.09 0.02 98.46 01:25:01 PM all 1.05 0.00 0.47 0.08 0.02 98.39 01:35:01 PM all 1.08 0.00 0.45 0.08 0.02 98.37 01:45:01 PM all 0.48 0.00 0.36 0.05 0.02 99.10 01:55:01 PM all 0.62 0.00 0.42 0.11 0.02 98.83 02:05:01 PM all 0.56 0.00 0.35 0.14 0.02 98.93 02:15:01 PM all 0.95 0.00 0.47 0.13 0.02 98.43 02:25:01 PM all 0.95 0.00 0.46 0.09 0.02 98.47 02:35:01 PM all 1.10 0.00 0.49 0.13 0.02 98.26 02:45:01 PM all 1.18 0.00 0.60 0.09 0.02 98.11 02:55:01 PM all 1.23 0.00 0.59 0.09 0.03 98.06 03:05:01 PM all 2.38 0.00 0.78 0.13 0.03 96.69 03:15:01 PM all 1.00 0.00 0.49 0.12 0.02 98.37 03:25:01 PM all 1.61 0.00 0.68 0.10 0.02 97.60 03:35:01 PM all 1.33 0.00 0.56 0.07 0.03 98.01 03:45:01 PM all 1.05 0.00 0.48 0.11 0.02 98.34 03:55:01 PM all 1.03 0.00 0.51 0.10 0.02 98.35 04:05:01 PM all 1.31 0.00 0.52 0.10 0.02 98.06 04:15:01 PM all 0.97 0.00 0.50 0.10 0.02 98.40 04:25:01 PM all 1.18 0.00 0.56 0.13 0.02 98.11 04:35:01 PM all 1.23 0.00 0.52 10.06 0.02 88.17 <-- trouble starts! 04:45:01 PM all 1.42 0.00 0.89 49.33 0.02 48.33 04:55:01 PM all 1.28 0.00 0.74 49.12 0.02 48.84 05:05:01 PM all 1.58 0.00 2.48 50.41 0.04 45.49 05:15:01 PM all 0.86 0.00 0.39 49.43 0.02 49.29