Seeing this bug on Ubuntu 20.04 and Ubuntu 22.04 as well, with both normal and HWE kernels. To add some more information, this bug seems to randomly appear during the initial RAID 6 creation process as well, where the array is mounted but completely empty and not accessed - so it's likely to originate within the mdadm resync process itself, unrelated to other system I/O operations. Around 1 in 15 arrays will "freeze" during the initial resync process in my experience, so it is not that uncommon unfortunately. The symptoms are always the same - at some point during resync, the speeds will radically drop to single MB/s levels and continue degrading over time until "echo active > /sys/block/mdX/md/array_state" is issued. A minute or two after running that, the speeds ramp back up to normal. This bug seems unrelated to hardware configuration, as I've seen it happen across multiple systems with different CPU vendors, HBA models and with different HDD sizes and vendors. Systems which were previously stable under Ubuntu 18.04 started exhibiting freezes after upgrading to 20.04 as well. It would also seem disabling mdcheck_start and mdcheck_continue is not necessarily the magical bullet in fixing this, it certainly doesn't seem to help with freezing during the initial resync. I have also seen instances of mdadm scheduled resyncs freezing when triggered using the old cronjob method, with both systemd services and timers masked off. Dmesg from a freshly installed system where the initial resync "froze" approximately 15 hours after the array was created: mdadm --create /dev/md1 --level=6 --raid-devices=6 /dev/sd[cdefgh] mkfs.ext4 /dev/md1 mount -o errors=remount-ro /dev/md1 /srv [Jul 4 00:18] md/raid:md1: not clean -- starting background reconstruction [ +0.000062] md/raid:md1: device sdh operational as raid disk 5 [ +0.000002] md/raid:md1: device sdg operational as raid disk 4 [ +0.000001] md/raid:md1: device sdf operational as raid disk 3 [ +0.000001] md/raid:md1: device sde operational as raid disk 2 [ +0.000001] md/raid:md1: device sdd operational as raid disk 1 [ +0.000001] md/raid:md1: device sdc operational as raid disk 0 [ +0.002104] md/raid:md1: raid level 6 active with 6 out of 6 devices, algorithm 2 [ +0.048241] md1: detected capacity change from 0 to 72000290684928 [ +0.000066] md: resync of RAID array md1 [Jul 4 00:32] EXT4-fs (md1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [Jul 4 02:39] perf: interrupt took too long (2522 > 2500), lowering kernel.perf_event_max_sample_rate to 79250 [Jul 4 04:36] perf: interrupt took too long (3155 > 3152), lowering kernel.perf_event_max_sample_rate to 63250 [Jul 4 15:22] INFO: task md1_raid6:5688 blocked for more than 120 seconds. [ +0.000059] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000032] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000045] md1_raid6 D 0 5688 2 0x80004000 [ +0.000004] Call Trace: [ +0.000011] __schedule+0x2e3/0x740 [ +0.000005] schedule+0x42/0xb0 [ +0.000011] raid5d+0x3e6/0x5f0 [raid456] [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000005] ? schedule_timeout+0x92/0x160 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] kthread+0x104/0x140 [ +0.000003] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000005] INFO: task md1_resync:5724 blocked for more than 120 seconds. [ +0.000039] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000044] md1_resync D 0 5724 2 0x80004000 [ +0.000002] Call Trace: [ +0.000005] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000007] raid5_get_active_stripe+0x459/0x610 [raid456] [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_sync_request+0x38b/0x3b0 [raid456] [ +0.000004] ? cpumask_next+0x1b/0x20 [ +0.000003] ? is_mddev_idle+0xc1/0x11e [ +0.000004] md_do_sync.cold+0x3ef/0x992 [ +0.000005] ? sched_clock+0x9/0x10 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000004] kthread+0x104/0x140 [ +0.000002] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000003] ret_from_fork+0x1f/0x40 [ +0.000003] INFO: task jbd2/md1-8:6099 blocked for more than 120 seconds. [ +0.000038] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000043] jbd2/md1-8 D 0 6099 2 0x80004000 [ +0.000001] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] ? __wake_up_common_lock+0x8a/0xc0 [ +0.000004] schedule+0x42/0xb0 [ +0.000005] jbd2_journal_commit_transaction+0x24e/0x18b0 [ +0.000004] ? dequeue_entity+0x118/0x460 [ +0.000002] ? record_times+0x1b/0x90 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000003] kjournald2+0xb6/0x280 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? commit_timeout+0x20/0x20 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task ext4lazyinit:6101 blocked for more than 120 seconds. [ +0.000038] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000043] ext4lazyinit D 0 6101 2 0x80004000 [ +0.000002] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] schedule+0x42/0xb0 [ +0.000004] md_write_start+0x151/0x230 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_make_request+0x88/0x870 [raid456] [ +0.000002] ? newidle_balance+0x231/0x3e0 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] ? __switch_to_asm+0x42/0x80 [ +0.000002] ? __switch_to_asm+0x5a/0x80 [ +0.000003] md_handle_request+0x123/0x1a0 [ +0.000004] md_make_request+0x92/0x1b0 [ +0.000003] generic_make_request+0xcf/0x320 [ +0.000004] ? bio_alloc_bioset+0x145/0x200 [ +0.000003] submit_bio+0x48/0x1d0 [ +0.000003] ? bio_add_page+0x6a/0x90 [ +0.000003] submit_bio_wait+0x58/0x90 [ +0.000004] blkdev_issue_zeroout+0x13f/0x220 [ +0.000006] ext4_init_inode_table+0x203/0x380 [ +0.000004] ext4_lazyinit_thread.part.0+0x2be/0x3c0 [ +0.000003] ext4_lazyinit_thread+0x20/0x30 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? ext4_lazyinit_thread.part.0+0x3c0/0x3c0 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000003] ret_from_fork+0x1f/0x40 [Jul 4 15:24] INFO: task md1_raid6:5688 blocked for more than 241 seconds. [ +0.000046] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000040] md1_raid6 D 0 5688 2 0x80004000 [ +0.000004] Call Trace: [ +0.000009] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000010] raid5d+0x3e6/0x5f0 [raid456] [ +0.000005] ? try_to_del_timer_sync+0x54/0x80 [ +0.000004] ? schedule_timeout+0x92/0x160 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] md_thread+0x97/0x160 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] kthread+0x104/0x140 [ +0.000003] ? md_start_sync+0x60/0x60 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task md1_resync:5724 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000029] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000039] md1_resync D 0 5724 2 0x80004000 [ +0.000002] Call Trace: [ +0.000004] __schedule+0x2e3/0x740 [ +0.000003] schedule+0x42/0xb0 [ +0.000007] raid5_get_active_stripe+0x459/0x610 [raid456] [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000007] raid5_sync_request+0x38b/0x3b0 [raid456] [ +0.000003] ? cpumask_next+0x1b/0x20 [ +0.000004] ? is_mddev_idle+0xc1/0x11e [ +0.000003] md_do_sync.cold+0x3ef/0x992 [ +0.000004] ? sched_clock+0x9/0x10 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] md_thread+0x97/0x160 [ +0.000003] kthread+0x104/0x140 [ +0.000002] ? md_start_sync+0x60/0x60 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task jbd2/md1-8:6099 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000038] jbd2/md1-8 D 0 6099 2 0x80004000 [ +0.000002] Call Trace: [ +0.000003] __schedule+0x2e3/0x740 [ +0.000003] ? __wake_up_common_lock+0x8a/0xc0 [ +0.000003] schedule+0x42/0xb0 [ +0.000005] jbd2_journal_commit_transaction+0x24e/0x18b0 [ +0.000003] ? dequeue_entity+0x118/0x460 [ +0.000003] ? record_times+0x1b/0x90 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000004] ? try_to_del_timer_sync+0x54/0x80 [ +0.000004] kjournald2+0xb6/0x280 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000003] kthread+0x104/0x140 [ +0.000003] ? commit_timeout+0x20/0x20 [ +0.000002] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40 [ +0.000004] INFO: task ext4lazyinit:6101 blocked for more than 241 seconds. [ +0.000035] Not tainted 5.4.0-153-generic #170-Ubuntu [ +0.000028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ +0.000039] ext4lazyinit D 0 6101 2 0x80004000 [ +0.000001] Call Trace: [ +0.000003] __schedule+0x2e3/0x740 [ +0.000004] schedule+0x42/0xb0 [ +0.000003] md_write_start+0x151/0x230 [ +0.000003] ? __wake_up_pollfree+0x40/0x40 [ +0.000006] raid5_make_request+0x88/0x870 [raid456] [ +0.000002] ? newidle_balance+0x231/0x3e0 [ +0.000004] ? __wake_up_pollfree+0x40/0x40 [ +0.000002] ? __switch_to_asm+0x42/0x80 [ +0.000002] ? __switch_to_asm+0x5a/0x80 [ +0.000003] md_handle_request+0x123/0x1a0 [ +0.000003] md_make_request+0x92/0x1b0 [ +0.000004] generic_make_request+0xcf/0x320 [ +0.000003] ? bio_alloc_bioset+0x145/0x200 [ +0.000003] submit_bio+0x48/0x1d0 [ +0.000002] ? bio_add_page+0x6a/0x90 [ +0.000003] submit_bio_wait+0x58/0x90 [ +0.000004] blkdev_issue_zeroout+0x13f/0x220 [ +0.000005] ext4_init_inode_table+0x203/0x380 [ +0.000004] ext4_lazyinit_thread.part.0+0x2be/0x3c0 [ +0.000003] ext4_lazyinit_thread+0x20/0x30 [ +0.000003] kthread+0x104/0x140 [ +0.000002] ? ext4_lazyinit_thread.part.0+0x3c0/0x3c0 [ +0.000003] ? kthread_park+0x90/0x90 [ +0.000002] ret_from_fork+0x1f/0x40