On Ubuntu 22.04, 5.15.0-83-generic #92-Ubuntu - our storage system ran into this bug. mdcheck ran for the scheduled 1st day of the month and then hung 6 hours later.


Oct  1 06:52:13 server1 systemd[1]: Starting MD array scrubbing...
Oct  1 06:52:13 server1 root: mdcheck start checking /dev/md0
Oct  1 06:52:13 server1 kernel: [2129098.393495] md: data-check of RAID array md0


Oct  1 12:57:49 server1 kernel: [2151034.623372] INFO: task dmcrypt_write/2:1783 blocked for more than 241 seconds.
Oct  1 12:57:49 server1 kernel: [2151034.623446]       Tainted: G S                5.15.0-83-generic #92-Ubuntu
Oct  1 12:57:49 server1 kernel: [2151034.623498] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 12:57:49 server1 kernel: [2151034.623559] task:dmcrypt_write/2 state:D stack:    0 pid: 1783 ppid:     2 flags:0x00004000
Oct  1 12:57:49 server1 kernel: [2151034.623566] Call Trace:
Oct  1 12:57:49 server1 kernel: [2151034.623570]  <TASK>
Oct  1 12:57:49 server1 kernel: [2151034.623574]  __schedule+0x24e/0x590
Oct  1 12:57:49 server1 kernel: [2151034.623585]  ? __schedule+0x256/0x590
Oct  1 12:57:49 server1 kernel: [2151034.623590]  schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.623596]  md_write_start.part.0+0x174/0x220
Oct  1 12:57:49 server1 kernel: [2151034.623601]  ? wait_woken+0x70/0x70
Oct  1 12:57:49 server1 kernel: [2151034.623610]  md_write_start+0x14/0x30
Oct  1 12:57:49 server1 kernel: [2151034.623615]  raid5_make_request+0x77/0x540 [raid456]
Oct  1 12:57:49 server1 kernel: [2151034.623633]  ? cgroup_rstat_updated+0x11c/0x1e0
Oct  1 12:57:49 server1 kernel: [2151034.623642]  ? wait_woken+0x70/0x70
Oct  1 12:57:49 server1 kernel: [2151034.623648]  md_handle_request+0x12d/0x1b0
Oct  1 12:57:49 server1 kernel: [2151034.623657]  ? submit_bio_checks+0x1a5/0x560
Oct  1 12:57:49 server1 kernel: [2151034.623664]  md_submit_bio+0x76/0xc0
Oct  1 12:57:49 server1 kernel: [2151034.623670]  __submit_bio+0x1a5/0x220
Oct  1 12:57:49 server1 kernel: [2151034.623675]  ? psi_task_switch+0xc6/0x220
Oct  1 12:57:49 server1 kernel: [2151034.623682]  __submit_bio_noacct+0x85/0x200
Oct  1 12:57:49 server1 kernel: [2151034.623687]  submit_bio_noacct+0x4e/0x120
Oct  1 12:57:49 server1 kernel: [2151034.623691]  ? schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.623698]  dmcrypt_write+0x104/0x130 [dm_crypt]
Oct  1 12:57:49 server1 kernel: [2151034.623708]  ? crypt_ctr+0x600/0x600 [dm_crypt]
Oct  1 12:57:49 server1 kernel: [2151034.623715]  kthread+0x12a/0x150
Oct  1 12:57:49 server1 kernel: [2151034.623723]  ? set_kthread_struct+0x50/0x50
Oct  1 12:57:49 server1 kernel: [2151034.623730]  ret_from_fork+0x22/0x30
Oct  1 12:57:49 server1 kernel: [2151034.623739]  </TASK>
Oct  1 12:57:49 server1 kernel: [2151034.623778] INFO: task mdcheck:2323903 blocked for more than 241 seconds.
Oct  1 12:57:49 server1 kernel: [2151034.623833]       Tainted: G S                5.15.0-83-generic #92-Ubuntu
Oct  1 12:57:49 server1 kernel: [2151034.625482] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 12:57:49 server1 kernel: [2151034.627098] task:mdcheck         state:D stack:    0 pid:2323903 ppid:     1 flags:0x00000002
Oct  1 12:57:49 server1 kernel: [2151034.627104] Call Trace:
Oct  1 12:57:49 server1 kernel: [2151034.627106]  <TASK>
Oct  1 12:57:49 server1 kernel: [2151034.627109]  __schedule+0x24e/0x590
Oct  1 12:57:49 server1 kernel: [2151034.627114]  ? select_idle_sibling+0x2b/0xa60
Oct  1 12:57:49 server1 kernel: [2151034.627124]  schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.627129]  schedule_timeout+0x103/0x140
Oct  1 12:57:49 server1 kernel: [2151034.627135]  ? ttwu_queue_wakelist+0x131/0x1c0
Oct  1 12:57:49 server1 kernel: [2151034.627142]  __wait_for_common+0xae/0x150
Oct  1 12:57:49 server1 kernel: [2151034.627148]  ? usleep_range_state+0x90/0x90
Oct  1 12:57:49 server1 kernel: [2151034.627155]  wait_for_completion+0x24/0x30
Oct  1 12:57:49 server1 kernel: [2151034.627160]  kthread_stop+0x6d/0x170
Oct  1 12:57:49 server1 kernel: [2151034.627168]  md_unregister_thread+0x44/0x90
Oct  1 12:57:49 server1 kernel: [2151034.627172]  md_reap_sync_thread+0x24/0x230
Oct  1 12:57:49 server1 kernel: [2151034.627177]  action_store+0x16f/0x300
Oct  1 12:57:49 server1 kernel: [2151034.627182]  md_attr_store+0x95/0xf0
Oct  1 12:57:49 server1 kernel: [2151034.627187]  sysfs_kf_write+0x3e/0x50
Oct  1 12:57:49 server1 kernel: [2151034.627194]  kernfs_fop_write_iter+0x13b/0x1c0
Oct  1 12:57:49 server1 kernel: [2151034.627199]  new_sync_write+0x114/0x1a0
Oct  1 12:57:49 server1 kernel: [2151034.627207]  vfs_write+0x1d5/0x270
Oct  1 12:57:49 server1 kernel: [2151034.627212]  ksys_write+0x67/0xf0
Oct  1 12:57:49 server1 kernel: [2151034.627219]  __x64_sys_write+0x19/0x20
Oct  1 12:57:49 server1 kernel: [2151034.627225]  do_syscall_64+0x5c/0xc0
Oct  1 12:57:49 server1 kernel: [2151034.627232]  ? do_syscall_64+0x69/0xc0
Oct  1 12:57:49 server1 kernel: [2151034.627258]  ? do_syscall_64+0x69/0xc0
Oct  1 12:57:49 server1 kernel: [2151034.627263]  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Oct  1 12:57:49 server1 kernel: [2151034.627271] RIP: 0033:0x7fb3bd5e8a37
Oct  1 12:57:49 server1 kernel: [2151034.627276] RSP: 002b:00007ffc02410e18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Oct  1 12:57:49 server1 kernel: [2151034.627282] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007fb3bd5e8a37
Oct  1 12:57:49 server1 kernel: [2151034.627285] RDX: 0000000000000005 RSI: 0000564ec7e41690 RDI: 0000000000000001
Oct  1 12:57:49 server1 kernel: [2151034.627288] RBP: 0000564ec7e41690 R08: 00007fb3bd6a5460 R09: 000000007fffffff
Oct  1 12:57:49 server1 kernel: [2151034.627291] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
Oct  1 12:57:49 server1 kernel: [2151034.627294] R13: 00007fb3bd6ee780 R14: 00007fb3bd6ea600 R15: 00007fb3bd6e9a00
Oct  1 12:57:49 server1 kernel: [2151034.627298]  </TASK>
Oct  1 12:57:49 server1 kernel: [2151034.627300] INFO: task md0_resync:2323914 blocked for more than 241 seconds.
Oct  1 12:57:49 server1 kernel: [2151034.628914]       Tainted: G S                5.15.0-83-generic #92-Ubuntu
Oct  1 12:57:49 server1 kernel: [2151034.630539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 12:57:49 server1 kernel: [2151034.632209] task:md0_resync      state:D stack:    0 pid:2323914 ppid:     2 flags:0x00004000
Oct  1 12:57:49 server1 kernel: [2151034.632216] Call Trace:
Oct  1 12:57:49 server1 kernel: [2151034.632218]  <TASK>
Oct  1 12:57:49 server1 kernel: [2151034.632220]  __schedule+0x24e/0x590
Oct  1 12:57:49 server1 kernel: [2151034.632226]  ? __wake_up_common_lock+0x8a/0xc0
Oct  1 12:57:49 server1 kernel: [2151034.632232]  schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.632238]  raid5_get_active_stripe+0x30e/0x3c0 [raid456]
Oct  1 12:57:49 server1 kernel: [2151034.632253]  ? wait_woken+0x70/0x70
Oct  1 12:57:49 server1 kernel: [2151034.632260]  raid5_sync_request+0x38c/0x3b0 [raid456]
Oct  1 12:57:49 server1 kernel: [2151034.632274]  ? __bpf_trace_tick_stop+0x20/0x20
Oct  1 12:57:49 server1 kernel: [2151034.632281]  md_do_sync.cold+0x42f/0xa17
Oct  1 12:57:49 server1 kernel: [2151034.632287]  ? wait_woken+0x70/0x70
Oct  1 12:57:49 server1 kernel: [2151034.632295]  md_thread+0xad/0x170
Oct  1 12:57:49 server1 kernel: [2151034.632305]  ? md_write_inc+0x60/0x60
Oct  1 12:57:49 server1 kernel: [2151034.632312]  kthread+0x12a/0x150
Oct  1 12:57:49 server1 kernel: [2151034.632320]  ? set_kthread_struct+0x50/0x50
Oct  1 12:57:49 server1 kernel: [2151034.632327]  ret_from_fork+0x22/0x30
Oct  1 12:57:49 server1 kernel: [2151034.632336]  </TASK>
Oct  1 12:57:49 server1 kernel: [2151034.632342] INFO: task kworker/7:1:2326463 blocked for more than 241 seconds.
Oct  1 12:57:49 server1 kernel: [2151034.633989]       Tainted: G S                5.15.0-83-generic #92-Ubuntu
Oct  1 12:57:49 server1 kernel: [2151034.635672] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 12:57:49 server1 kernel: [2151034.637344] task:kworker/7:1     state:D stack:    0 pid:2326463 ppid:     2 flags:0x00004000
Oct  1 12:57:49 server1 kernel: [2151034.637351] Workqueue: xfs-sync/dm-0 xfs_log_worker [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.637562] Call Trace:
Oct  1 12:57:49 server1 kernel: [2151034.637564]  <TASK>
Oct  1 12:57:49 server1 kernel: [2151034.637566]  __schedule+0x24e/0x590
Oct  1 12:57:49 server1 kernel: [2151034.637573]  schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.637578]  schedule_timeout+0x103/0x140
Oct  1 12:57:49 server1 kernel: [2151034.637585]  ? ttwu_queue_wakelist+0x131/0x1c0
Oct  1 12:57:49 server1 kernel: [2151034.637591]  __wait_for_common+0xae/0x150
Oct  1 12:57:49 server1 kernel: [2151034.637596]  ? usleep_range_state+0x90/0x90
Oct  1 12:57:49 server1 kernel: [2151034.637603]  wait_for_completion+0x24/0x30
Oct  1 12:57:49 server1 kernel: [2151034.637609]  flush_workqueue+0x133/0x3e0
Oct  1 12:57:49 server1 kernel: [2151034.637615]  xlog_cil_push_now.isra.0+0x94/0xa0 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.637802]  xlog_cil_force_seq+0x78/0x2b0 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.637987]  ? psi_task_switch+0xc6/0x220
Oct  1 12:57:49 server1 kernel: [2151034.637994]  xfs_log_force+0x9e/0x260 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.638180]  xfs_log_worker+0x39/0x90 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.638364]  process_one_work+0x22b/0x3d0
Oct  1 12:57:49 server1 kernel: [2151034.638370]  worker_thread+0x53/0x420
Oct  1 12:57:49 server1 kernel: [2151034.638374]  ? process_one_work+0x3d0/0x3d0
Oct  1 12:57:49 server1 kernel: [2151034.638378]  kthread+0x12a/0x150
Oct  1 12:57:49 server1 kernel: [2151034.638385]  ? set_kthread_struct+0x50/0x50
Oct  1 12:57:49 server1 kernel: [2151034.638392]  ret_from_fork+0x22/0x30
Oct  1 12:57:49 server1 kernel: [2151034.638401]  </TASK>
Oct  1 12:57:49 server1 kernel: [2151034.638411] INFO: task kworker/u114:38:2342034 blocked for more than 241 seconds.
Oct  1 12:57:49 server1 kernel: [2151034.640574]       Tainted: G S                5.15.0-83-generic #92-Ubuntu
Oct  1 12:57:49 server1 kernel: [2151034.642748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct  1 12:57:49 server1 kernel: [2151034.644960] task:kworker/u114:38 state:D stack:    0 pid:2342034 ppid:     2 flags:0x00004000
Oct  1 12:57:49 server1 kernel: [2151034.644967] Workqueue: xfs-cil/dm-0 xlog_cil_push_work [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.645157] Call Trace:
Oct  1 12:57:49 server1 kernel: [2151034.645159]  <TASK>
Oct  1 12:57:49 server1 kernel: [2151034.645161]  __schedule+0x24e/0x590
Oct  1 12:57:49 server1 kernel: [2151034.645168]  ? submit_bio+0x4a/0x130
Oct  1 12:57:49 server1 kernel: [2151034.645173]  schedule+0x69/0x110
Oct  1 12:57:49 server1 kernel: [2151034.645179]  xlog_state_get_iclog_space+0x120/0x370 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.645367]  ? wake_up_q+0x90/0x90
Oct  1 12:57:49 server1 kernel: [2151034.645375]  xlog_write+0x148/0x680 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.645563]  xlog_cil_push_work+0x34b/0x470 [xfs]
Oct  1 12:57:49 server1 kernel: [2151034.645750]  process_one_work+0x22b/0x3d0
Oct  1 12:57:49 server1 kernel: [2151034.645755]  worker_thread+0x53/0x420
Oct  1 12:57:49 server1 kernel: [2151034.645759]  ? process_one_work+0x3d0/0x3d0
Oct  1 12:57:49 server1 kernel: [2151034.645763]  kthread+0x12a/0x150
Oct  1 12:57:49 server1 kernel: [2151034.645770]  ? set_kthread_struct+0x50/0x50
Oct  1 12:57:49 server1 kernel: [2151034.645776]  ret_from_fork+0x22/0x30
Oct  1 12:57:49 server1 kernel: [2151034.645785]  </TASK>