I think I'm running into this too.  

Raid 6 array with 4 drives.


Linux pkitty 5.13.0-52-generic #59~20.04.1-Ubuntu SMP Thu Jun 16 21:21:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I have all filesystems unmounted (so there's no other activity that should be occurring on the drives).

Once it hangs, even trying to do 

echo idle /sys/block/md0/md/sync_action

hangs and the entire array locks up.

It's not a matter of if it will happen, but when.

Is there any way to backport the fixes?  5.19 is a long way away.

kern.log snippet:


ul  5 19:16:36 pkitty kernel: [ 4471.805561] INFO: task scsi_eh_1:276 blocked for more than 120 seconds.
Jul  5 19:16:36 pkitty kernel: [ 4471.805573]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:16:36 pkitty kernel: [ 4471.805577] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:16:36 pkitty kernel: [ 4471.805580] task:scsi_eh_1       state:D stack:    0 pid:  276 ppid:     2 flags:0x00004000
Jul  5 19:16:36 pkitty kernel: [ 4471.805588] Call Trace:
Jul  5 19:16:36 pkitty kernel: [ 4471.805591]  <TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.805596]  __schedule+0x2ee/0x900
Jul  5 19:16:36 pkitty kernel: [ 4471.805605]  schedule+0x4f/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.805609]  schedule_preempt_disabled+0xe/0x10
Jul  5 19:16:36 pkitty kernel: [ 4471.805614]  __mutex_lock.isra.0+0x183/0x4d0
Jul  5 19:16:36 pkitty kernel: [ 4471.805618]  ? dequeue_entity+0xdb/0x410
Jul  5 19:16:36 pkitty kernel: [ 4471.805627]  ? blk_mq_find_and_get_req+0x4f/0x90
Jul  5 19:16:36 pkitty kernel: [ 4471.805634]  __mutex_lock_slowpath+0x13/0x20
Jul  5 19:16:36 pkitty kernel: [ 4471.805639]  mutex_lock+0x32/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805645]  device_reset+0x22/0x50 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805654]  scsi_eh_ready_devs+0x580/0xa50
Jul  5 19:16:36 pkitty kernel: [ 4471.805683]  kthread+0x12b/0x150
Jul  5 19:16:36 pkitty kernel: [ 4471.805687]  ? set_kthread_struct+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805692]  ret_from_fork+0x22/0x30
Jul  5 19:16:36 pkitty kernel: [ 4471.805701]  </TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.805704] INFO: task usb-storage:278 blocked for more than 241 seconds.
Jul  5 19:16:36 pkitty kernel: [ 4471.805708]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:16:36 pkitty kernel: [ 4471.805711] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:16:36 pkitty kernel: [ 4471.805713] task:usb-storage     state:D stack:    0 pid:  278 ppid:     2 flags:0x00004000
Jul  5 19:16:36 pkitty kernel: [ 4471.805718] Call Trace:
Jul  5 19:16:36 pkitty kernel: [ 4471.805720]  <TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.805722]  __schedule+0x2ee/0x900
Jul  5 19:16:36 pkitty kernel: [ 4471.805727]  ? usleep_range_state+0x90/0x90
Jul  5 19:16:36 pkitty kernel: [ 4471.805733]  schedule+0x4f/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.805736]  schedule_timeout+0x202/0x290
Jul  5 19:16:36 pkitty kernel: [ 4471.805742]  ? xhci_urb_enqueue+0x1e4/0x520
Jul  5 19:16:36 pkitty kernel: [ 4471.805746]  ? usleep_range_state+0x90/0x90
Jul  5 19:16:36 pkitty kernel: [ 4471.805751]  __wait_for_common+0xba/0x160
Jul  5 19:16:36 pkitty kernel: [ 4471.805756]  wait_for_completion+0x24/0x30
Jul  5 19:16:36 pkitty kernel: [ 4471.805760]  usb_sg_wait+0xea/0x170
Jul  5 19:16:36 pkitty kernel: [ 4471.805767]  usb_stor_bulk_transfer_sglist+0x97/0xe0 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805776]  usb_stor_bulk_srb+0x3d/0x70 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805783]  usb_stor_Bulk_transport+0x177/0x410 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805790]  ? schedule_timeout+0x202/0x290
Jul  5 19:16:36 pkitty kernel: [ 4471.805796]  usb_stor_invoke_transport+0x3e/0x520 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805804]  ? usleep_range_state+0x90/0x90
Jul  5 19:16:36 pkitty kernel: [ 4471.805809]  ? __wait_for_common+0xfb/0x160
Jul  5 19:16:36 pkitty kernel: [ 4471.805813]  ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
Jul  5 19:16:36 pkitty kernel: [ 4471.805821]  usb_stor_transparent_scsi_command+0xe/0x10 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805828]  usb_stor_control_thread+0x19b/0x2a0 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805836]  ? storage_probe+0x2b0/0x2b0 [usb_storage]
Jul  5 19:16:36 pkitty kernel: [ 4471.805844]  kthread+0x12b/0x150
Jul  5 19:16:36 pkitty kernel: [ 4471.805848]  ? set_kthread_struct+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805852]  ret_from_fork+0x22/0x30
Jul  5 19:16:36 pkitty kernel: [ 4471.805861]  </TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.805865] INFO: task md0_raid6:419 blocked for more than 241 seconds.
Jul  5 19:16:36 pkitty kernel: [ 4471.805869]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:16:36 pkitty kernel: [ 4471.805872] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:16:36 pkitty kernel: [ 4471.805874] task:md0_raid6       state:D stack:    0 pid:  419 ppid:     2 flags:0x00004000
Jul  5 19:16:36 pkitty kernel: [ 4471.805878] Call Trace:
Jul  5 19:16:36 pkitty kernel: [ 4471.805880]  <TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.805882]  __schedule+0x2ee/0x900
Jul  5 19:16:36 pkitty kernel: [ 4471.805885]  ? blk_flush_plug_list+0xe3/0x110
Jul  5 19:16:36 pkitty kernel: [ 4471.805891]  schedule+0x4f/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.805895]  io_schedule+0x16/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805899]  blk_mq_get_tag+0x11d/0x270
Jul  5 19:16:36 pkitty kernel: [ 4471.805903]  ? __wake_up_pollfree+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805910]  __blk_mq_alloc_request+0xec/0x120
Jul  5 19:16:36 pkitty kernel: [ 4471.805919]  blk_mq_submit_bio+0x108/0x590
Jul  5 19:16:36 pkitty kernel: [ 4471.805924]  submit_bio_noacct+0x446/0x510
Jul  5 19:16:36 pkitty kernel: [ 4471.805930]  ops_run_io+0x724/0xcb0 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.805943]  handle_stripe+0x59f/0x1360 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.805958]  handle_active_stripes.isra.0+0x3c5/0x5a0 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.805970]  raid5d+0x35d/0x550 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.805981]  md_thread+0xb1/0x170
Jul  5 19:16:36 pkitty kernel: [ 4471.805986]  ? __wake_up_pollfree+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.805992]  ? md_start_sync+0x60/0x60
Jul  5 19:16:36 pkitty kernel: [ 4471.805996]  kthread+0x12b/0x150
Jul  5 19:16:36 pkitty kernel: [ 4471.806000]  ? set_kthread_struct+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806005]  ret_from_fork+0x22/0x30
Jul  5 19:16:36 pkitty kernel: [ 4471.806013]  </TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.806031] INFO: task pool-udisksd:6829 blocked for more than 241 seconds.
Jul  5 19:16:36 pkitty kernel: [ 4471.806035]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:16:36 pkitty kernel: [ 4471.806038] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:16:36 pkitty kernel: [ 4471.806040] task:pool-udisksd    state:D stack:    0 pid: 6829 ppid:     1 flags:0x00000000
Jul  5 19:16:36 pkitty kernel: [ 4471.806045] Call Trace:
Jul  5 19:16:36 pkitty kernel: [ 4471.806047]  <TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.806049]  __schedule+0x2ee/0x900
Jul  5 19:16:36 pkitty kernel: [ 4471.806054]  schedule+0x4f/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.806057]  io_schedule+0x16/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806061]  blk_mq_get_tag+0x11d/0x270
Jul  5 19:16:36 pkitty kernel: [ 4471.806065]  ? __wake_up_pollfree+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806072]  __blk_mq_alloc_request+0xec/0x120
Jul  5 19:16:36 pkitty kernel: [ 4471.806079]  blk_mq_alloc_request+0x7c/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.806085]  blk_get_request+0x27/0x60
Jul  5 19:16:36 pkitty kernel: [ 4471.806093]  sg_io+0xa0/0x3f0
Jul  5 19:16:36 pkitty kernel: [ 4471.806096]  ? _copy_from_user+0x2b/0x60
Jul  5 19:16:36 pkitty kernel: [ 4471.806101]  ? get_sg_io_hdr+0xc2/0xf0
Jul  5 19:16:36 pkitty kernel: [ 4471.806105]  scsi_cmd_ioctl+0x24c/0x2a0
Jul  5 19:16:36 pkitty kernel: [ 4471.806110]  scsi_cmd_blk_ioctl+0x51/0x70
Jul  5 19:16:36 pkitty kernel: [ 4471.806114]  sd_ioctl_common+0xbb/0x1f0
Jul  5 19:16:36 pkitty kernel: [ 4471.806119]  sd_ioctl+0x1c/0x50
Jul  5 19:16:36 pkitty kernel: [ 4471.806124]  blkdev_ioctl+0x11c/0x250
Jul  5 19:16:36 pkitty kernel: [ 4471.806128]  ? __fget_light+0xce/0xf0
Jul  5 19:16:36 pkitty kernel: [ 4471.806136]  block_ioctl+0x3d/0x50
Jul  5 19:16:36 pkitty kernel: [ 4471.806140]  __x64_sys_ioctl+0x91/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.806146]  do_syscall_64+0x61/0xb0
Jul  5 19:16:36 pkitty kernel: [ 4471.806151]  ? sysvec_apic_timer_interrupt+0x4e/0x90
Jul  5 19:16:36 pkitty kernel: [ 4471.806157]  ? asm_sysvec_apic_timer_interrupt+0xa/0x20
Jul  5 19:16:36 pkitty kernel: [ 4471.806164]  entry_SYSCALL_64_after_hwframe+0x44/0xae
Jul  5 19:16:36 pkitty kernel: [ 4471.806170] RIP: 0033:0x7f52345bc3ab
Jul  5 19:16:36 pkitty kernel: [ 4471.806175] RSP: 002b:00007f523356e8e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Jul  5 19:16:36 pkitty kernel: [ 4471.806180] RAX: ffffffffffffffda RBX: 00007f523356ea80 RCX: 00007f52345bc3ab
Jul  5 19:16:36 pkitty kernel: [ 4471.806183] RDX: 00007f523356e900 RSI: 0000000000002285 RDI: 0000000000000012
Jul  5 19:16:36 pkitty kernel: [ 4471.806186] RBP: 0000000000001388 R08: 00007f523356eaa0 R09: 00007f523356f5e8
Jul  5 19:16:36 pkitty kernel: [ 4471.806189] R10: 00007f523356ea10 R11: 0000000000000246 R12: 00007f523356ec10
Jul  5 19:16:36 pkitty kernel: [ 4471.806191] R13: 0000000000000000 R14: 0000000000000012 R15: 00007f523356eaa0
Jul  5 19:16:36 pkitty kernel: [ 4471.806196]  </TASK>
Jul  5 19:16:36 pkitty kernel: [ 4471.806255] INFO: task md0_resync:5122 blocked for more than 241 seconds.
Jul  5 19:16:36 pkitty kernel: [ 4471.806259]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:16:36 pkitty kernel: [ 4471.806262] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:16:36 pkitty kernel: [ 4471.806272]  __schedule+0x2ee/0x900
Jul  5 19:16:36 pkitty kernel: [ 4471.806277]  schedule+0x4f/0xc0
Jul  5 19:16:36 pkitty kernel: [ 4471.806281]  raid5_get_active_stripe+0x449/0x5e0 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.806292]  ? __wake_up_pollfree+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806299]  raid5_sync_request+0x387/0x3b0 [raid456]
Jul  5 19:16:36 pkitty kernel: [ 4471.806311]  md_do_sync.cold+0x3ed/0x953
Jul  5 19:16:36 pkitty kernel: [ 4471.806317]  ? __wake_up_pollfree+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806325]  md_thread+0xb1/0x170
Jul  5 19:16:36 pkitty kernel: [ 4471.806330]  ? md_start_sync+0x60/0x60
Jul  5 19:16:36 pkitty kernel: [ 4471.806334]  kthread+0x12b/0x150
Jul  5 19:16:36 pkitty kernel: [ 4471.806338]  ? set_kthread_struct+0x40/0x40
Jul  5 19:16:36 pkitty kernel: [ 4471.806342]  ret_from_fork+0x22/0x30
Jul  5 19:16:36 pkitty kernel: [ 4471.806351]  </TASK>

[...]

Jul  5 19:18:37 pkitty kernel: [ 4592.635404] INFO: task scsi_eh_1:276 blocked for more than 241 seconds.
Jul  5 19:18:37 pkitty kernel: [ 4592.635412]       Tainted: G           OE     5.13.0-52-generic #59~20.04.1-Ubuntu
Jul  5 19:18:37 pkitty kernel: [ 4592.635415] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul  5 19:18:37 pkitty kernel: [ 4592.635417] task:scsi_eh_1       state:D stack:    0 pid:  276 ppid:     2 flags:0x00004000
Jul  5 19:18:37 pkitty kernel: [ 4592.635423] Call Trace:
Jul  5 19:18:37 pkitty kernel: [ 4592.635425]  <TASK>
Jul  5 19:18:37 pkitty kernel: [ 4592.635428]  __schedule+0x2ee/0x900
Jul  5 19:18:37 pkitty kernel: [ 4592.635435]  schedule+0x4f/0xc0
Jul  5 19:18:37 pkitty kernel: [ 4592.635437]  schedule_preempt_disabled+0xe/0x10
Jul  5 19:18:37 pkitty kernel: [ 4592.635440]  __mutex_lock.isra.0+0x183/0x4d0
Jul  5 19:18:37 pkitty kernel: [ 4592.635457]  mutex_lock+0x32/0x40
Jul  5 19:18:37 pkitty kernel: [ 4592.635462]  device_reset+0x22/0x50 [usb_storage]
Jul  5 19:18:37 pkitty kernel: [ 4592.635468]  scsi_eh_ready_devs+0x580/0xa50
Jul  5 19:18:37 pkitty kernel: [ 4592.635474]  ? __pm_runtime_resume+0x60/0x80
Jul  5 19:18:37 pkitty kernel: [ 4592.635477]  scsi_error_handler+0x446/0x520
Jul  5 19:18:37 pkitty kernel: [ 4592.635482]  ? scsi_eh_get_sense+0x230/0x230
Jul  5 19:18:37 pkitty kernel: [ 4592.635487]  kthread+0x12b/0x150
Jul  5 19:18:37 pkitty kernel: [ 4592.635490]  ? set_kthread_struct+0x40/0x40
Jul  5 19:18:37 pkitty kernel: [ 4592.635493]  ret_from_fork+0x22/0x30
Jul  5 19:18:37 pkitty kernel: [ 4592.635499]  </TASK>