[756484.534293] md: data-check of RAID array md0 [756484.628039] md: delaying data-check of md1 until md0 has finished (they share one or more physical units) [756493.808773] md: md0: data-check done. [756493.829760] md: data-check of RAID array md1 [778112.446410] md: md1: data-check interrupted. [810654.608102] md: data-check of RAID array md1 [832291.201064] md: md1: data-check interrupted. [899745.389485] md: data-check of RAID array md1 [921395.835305] md: md1: data-check interrupted. [921588.558834] INFO: task systemd-journal:376 blocked for more than 120 seconds. [921588.558846] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.558850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.558854] task:systemd-journal state:D stack: 0 pid: 376 ppid: 1 flags:0x00000220 [921588.558859] Call Trace: [921588.558864] __schedule+0x44c/0x8a0 [921588.558872] schedule+0x4f/0xc0 [921588.558876] md_write_start+0x150/0x240 [921588.558880] ? wait_woken+0x80/0x80 [921588.558886] raid5_make_request+0x88/0x890 [raid456] [921588.558898] ? wait_woken+0x80/0x80 [921588.558901] ? mempool_kmalloc+0x17/0x20 [921588.558904] md_handle_request+0x12d/0x1a0 [921588.558907] ? __part_start_io_acct+0x51/0xf0 [921588.558912] md_submit_bio+0xca/0x100 [921588.558915] submit_bio_noacct+0x112/0x4f0 [921588.558918] ? ext4_fc_reserve_space+0x110/0x230 [921588.558922] submit_bio+0x51/0x1a0 [921588.558925] ? _cond_resched+0x19/0x30 [921588.558928] ? kmem_cache_alloc+0x38e/0x440 [921588.558932] ? ext4_init_io_end+0x1f/0x50 [921588.558936] ext4_io_submit+0x4d/0x60 [921588.558940] ext4_writepages+0x2c6/0xcd0 [921588.558944] do_writepages+0x43/0xd0 [921588.558948] ? do_writepages+0x43/0xd0 [921588.558951] ? fault_dirty_shared_page+0xa5/0x110 [921588.558955] __filemap_fdatawrite_range+0xcc/0x110 [921588.558960] file_write_and_wait_range+0x74/0xc0 [921588.558962] ext4_sync_file+0xf5/0x350 [921588.558967] vfs_fsync_range+0x49/0x80 [921588.558970] do_fsync+0x3d/0x70 [921588.558973] __x64_sys_fsync+0x14/0x20 [921588.558976] do_syscall_64+0x38/0x90 [921588.558980] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.558984] RIP: 0033:0x7f4c97ee832b [921588.558987] RSP: 002b:00007ffdceb29e50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921588.558991] RAX: ffffffffffffffda RBX: 000055ced34b0fa0 RCX: 00007f4c97ee832b [921588.558993] RDX: 00007f4c97fc8000 RSI: 000055ced3487b70 RDI: 0000000000000021 [921588.558995] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007ffdceb29fa8 [921588.558996] R10: 00007f4c97d2c848 R11: 0000000000000293 R12: 00007ffdceb29fa8 [921588.558998] R13: 00007ffdceb29fa0 R14: 000055ced34b0fa0 R15: 000055ced34bcf90 [921588.559014] INFO: task mysqld:1505 blocked for more than 120 seconds. [921588.559018] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559025] task:mysqld state:D stack: 0 pid: 1505 ppid: 1 flags:0x00000000 [921588.559030] Call Trace: [921588.559032] __schedule+0x44c/0x8a0 [921588.559036] schedule+0x4f/0xc0 [921588.559040] md_write_start+0x150/0x240 [921588.559044] ? wait_woken+0x80/0x80 [921588.559047] raid5_make_request+0x88/0x890 [raid456] [921588.559056] ? wait_woken+0x80/0x80 [921588.559059] ? mempool_kmalloc+0x17/0x20 [921588.559062] md_handle_request+0x12d/0x1a0 [921588.559065] ? __part_start_io_acct+0x51/0xf0 [921588.559068] md_submit_bio+0xca/0x100 [921588.559071] submit_bio_noacct+0x112/0x4f0 [921588.559075] submit_bio+0x51/0x1a0 [921588.559077] ? _cond_resched+0x19/0x30 [921588.559081] ? kmem_cache_alloc+0x38e/0x440 [921588.559084] ? ext4_init_io_end+0x1f/0x50 [921588.559088] ext4_io_submit+0x4d/0x60 [921588.559091] ext4_writepages+0x2c6/0xcd0 [921588.559094] ? __schedule+0x454/0x8a0 [921588.559097] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921588.559100] ? timerqueue_del+0x24/0x50 [921588.559105] ? futex_wait+0x1ed/0x270 [921588.559109] do_writepages+0x43/0xd0 [921588.559112] ? do_writepages+0x43/0xd0 [921588.559115] ? futex_wake+0x7c/0x180 [921588.559118] __filemap_fdatawrite_range+0xcc/0x110 [921588.559123] file_write_and_wait_range+0x74/0xc0 [921588.559126] ext4_sync_file+0xf5/0x350 [921588.559129] vfs_fsync_range+0x49/0x80 [921588.559132] ? __fget_light+0x62/0x80 [921588.559136] do_fsync+0x3d/0x70 [921588.559139] __x64_sys_fsync+0x14/0x20 [921588.559142] do_syscall_64+0x38/0x90 [921588.559145] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.559148] RIP: 0033:0x7f36188d48db [921588.559150] RSP: 002b:00007f35f05487d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921588.559154] RAX: ffffffffffffffda RBX: 00007f360502bf50 RCX: 00007f36188d48db [921588.559156] RDX: 000d239200000000 RSI: 0000000000000000 RDI: 000000000000000a [921588.559158] RBP: 00007f35f0548a00 R08: 0000000000000000 R09: 00007f360502beb0 [921588.559160] R10: 00007f35f0548ccf R11: 0000000000000293 R12: 00007f36042e9610 [921588.559162] R13: 000000000000000a R14: 0000000000000b00 R15: 0000000000000000 [921588.559175] INFO: task mdcheck:40625 blocked for more than 120 seconds. [921588.559179] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559182] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559185] task:mdcheck state:D stack: 0 pid:40625 ppid: 1 flags:0x00000000 [921588.559189] Call Trace: [921588.559191] __schedule+0x44c/0x8a0 [921588.559196] schedule+0x4f/0xc0 [921588.559200] schedule_timeout+0x202/0x290 [921588.559203] ? do_truncate+0x78/0xd0 [921588.559216] ? try_to_wake_up+0x66/0x5c0 [921588.559220] wait_for_completion+0x94/0x100 [921588.559225] kthread_stop+0x71/0x170 [921588.559229] md_unregister_thread+0x47/0x80 [921588.559232] md_reap_sync_thread+0x24/0x200 [921588.559236] action_store+0x160/0x2e0 [921588.559239] md_attr_store+0x82/0xc0 [921588.559243] sysfs_kf_write+0x3e/0x50 [921588.559246] kernfs_fop_write_iter+0x138/0x1c0 [921588.559249] new_sync_write+0x117/0x1b0 [921588.559254] vfs_write+0x1ca/0x280 [921588.559257] ksys_write+0x67/0xe0 [921588.559260] __x64_sys_write+0x1a/0x20 [921588.559264] do_syscall_64+0x38/0x90 [921588.559267] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921588.559271] RIP: 0033:0x7f8c6d2791e7 [921588.559273] RSP: 002b:00007ffea5d97178 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [921588.559276] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f8c6d2791e7 [921588.559278] RDX: 0000000000000005 RSI: 000055b618a3eb40 RDI: 0000000000000001 [921588.559280] RBP: 000055b618a3eb40 R08: 000000000000000a R09: 0000000000000004 [921588.559282] R10: 000055b616e97017 R11: 0000000000000246 R12: 0000000000000005 [921588.559284] R13: 00007f8c6d3546a0 R14: 00007f8c6d3554a0 R15: 00007f8c6d3548a0 [921588.559287] INFO: task md1_resync:40644 blocked for more than 120 seconds. [921588.559291] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559294] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559297] task:md1_resync state:D stack: 0 pid:40644 ppid: 2 flags:0x00004000 [921588.559301] Call Trace: [921588.559303] __schedule+0x44c/0x8a0 [921588.559308] schedule+0x4f/0xc0 [921588.559311] md_do_sync.cold+0x8b7/0x953 [921588.559315] ? wait_woken+0x80/0x80 [921588.559319] ? kernel_sigaction+0x3b/0xd0 [921588.559323] md_thread+0xb1/0x170 [921588.559326] kthread+0x114/0x150 [921588.559329] ? md_start_sync+0x60/0x60 [921588.559332] ? kthread_park+0x90/0x90 [921588.559335] ret_from_fork+0x22/0x30 [921588.559341] INFO: task kworker/u16:0:41409 blocked for more than 120 seconds. [921588.559345] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921588.559348] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921588.559353] task:kworker/u16:0 state:D stack: 0 pid:41409 ppid: 2 flags:0x00004000 [921588.559357] Workqueue: writeback wb_workfn (flush-253:1) [921588.559361] Call Trace: [921588.559363] __schedule+0x44c/0x8a0 [921588.559367] schedule+0x4f/0xc0 [921588.559371] md_write_start+0x150/0x240 [921588.559375] ? wait_woken+0x80/0x80 [921588.559378] raid5_make_request+0x88/0x890 [raid456] [921588.559387] ? wait_woken+0x80/0x80 [921588.559390] ? mempool_kmalloc+0x17/0x20 [921588.559393] md_handle_request+0x12d/0x1a0 [921588.559396] ? __part_start_io_acct+0x51/0xf0 [921588.559399] md_submit_bio+0xca/0x100 [921588.559402] submit_bio_noacct+0x112/0x4f0 [921588.559405] submit_bio+0x51/0x1a0 [921588.559408] ? _cond_resched+0x19/0x30 [921588.559412] ? kmem_cache_alloc+0x38e/0x440 [921588.559415] ? ext4_init_io_end+0x1f/0x50 [921588.559419] ext4_io_submit+0x4d/0x60 [921588.559422] ext4_writepages+0x2c6/0xcd0 [921588.559425] ? update_group_capacity+0x2c/0x1e0 [921588.559430] do_writepages+0x43/0xd0 [921588.559433] ? do_writepages+0x43/0xd0 [921588.559436] __writeback_single_inode+0x44/0x2b0 [921588.559439] writeback_sb_inodes+0x22d/0x4c0 [921588.559443] __writeback_inodes_wb+0x56/0xf0 [921588.559446] wb_writeback+0x1ea/0x2a0 [921588.559450] wb_workfn+0x31b/0x490 [921588.559453] ? __update_idle_core+0x9b/0xb0 [921588.559456] process_one_work+0x220/0x3c0 [921588.559459] worker_thread+0x4d/0x3f0 [921588.559462] kthread+0x114/0x150 [921588.559465] ? process_one_work+0x3c0/0x3c0 [921588.559467] ? kthread_park+0x90/0x90 [921588.559471] ret_from_fork+0x22/0x30 [921709.391056] INFO: task jbd2/dm-1-8:319 blocked for more than 120 seconds. [921709.391068] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391073] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391077] task:jbd2/dm-1-8 state:D stack: 0 pid: 319 ppid: 2 flags:0x00004000 [921709.391083] Call Trace: [921709.391087] __schedule+0x44c/0x8a0 [921709.391094] schedule+0x4f/0xc0 [921709.391098] md_write_start+0x150/0x240 [921709.391103] ? wait_woken+0x80/0x80 [921709.391108] raid5_make_request+0x88/0x890 [raid456] [921709.391120] ? wait_woken+0x80/0x80 [921709.391123] ? mempool_kmalloc+0x17/0x20 [921709.391127] md_handle_request+0x12d/0x1a0 [921709.391130] ? __part_start_io_acct+0x51/0xf0 [921709.391134] md_submit_bio+0xca/0x100 [921709.391137] submit_bio_noacct+0x112/0x4f0 [921709.391141] submit_bio+0x51/0x1a0 [921709.391144] ? bio_add_page+0x6a/0x90 [921709.391147] submit_bh_wbc+0x195/0x1c0 [921709.391151] submit_bh+0x13/0x20 [921709.391154] jbd2_journal_commit_transaction+0x74b/0x18f0 [921709.391159] ? try_to_del_timer_sync+0x54/0x80 [921709.391164] kjournald2+0xb6/0x280 [921709.391168] ? wait_woken+0x80/0x80 [921709.391170] kthread+0x114/0x150 [921709.391174] ? commit_timeout+0x20/0x20 [921709.391177] ? kthread_park+0x90/0x90 [921709.391180] ret_from_fork+0x22/0x30 [921709.391186] INFO: task systemd-journal:376 blocked for more than 241 seconds. [921709.391191] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391195] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391199] task:systemd-journal state:D stack: 0 pid: 376 ppid: 1 flags:0x00000220 [921709.391204] Call Trace: [921709.391205] __schedule+0x44c/0x8a0 [921709.391210] schedule+0x4f/0xc0 [921709.391214] md_write_start+0x150/0x240 [921709.391218] ? wait_woken+0x80/0x80 [921709.391221] raid5_make_request+0x88/0x890 [raid456] [921709.391230] ? wait_woken+0x80/0x80 [921709.391233] ? mempool_kmalloc+0x17/0x20 [921709.391236] md_handle_request+0x12d/0x1a0 [921709.391239] ? __part_start_io_acct+0x51/0xf0 [921709.391242] md_submit_bio+0xca/0x100 [921709.391245] submit_bio_noacct+0x112/0x4f0 [921709.391248] ? ext4_fc_reserve_space+0x110/0x230 [921709.391252] submit_bio+0x51/0x1a0 [921709.391255] ? _cond_resched+0x19/0x30 [921709.391259] ? kmem_cache_alloc+0x38e/0x440 [921709.391262] ? ext4_init_io_end+0x1f/0x50 [921709.391266] ext4_io_submit+0x4d/0x60 [921709.391270] ext4_writepages+0x2c6/0xcd0 [921709.391274] do_writepages+0x43/0xd0 [921709.391278] ? do_writepages+0x43/0xd0 [921709.391281] ? fault_dirty_shared_page+0xa5/0x110 [921709.391285] __filemap_fdatawrite_range+0xcc/0x110 [921709.391290] file_write_and_wait_range+0x74/0xc0 [921709.391293] ext4_sync_file+0xf5/0x350 [921709.391297] vfs_fsync_range+0x49/0x80 [921709.391300] do_fsync+0x3d/0x70 [921709.391303] __x64_sys_fsync+0x14/0x20 [921709.391306] do_syscall_64+0x38/0x90 [921709.391310] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391314] RIP: 0033:0x7f4c97ee832b [921709.391317] RSP: 002b:00007ffdceb29e50 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921709.391321] RAX: ffffffffffffffda RBX: 000055ced34b0fa0 RCX: 00007f4c97ee832b [921709.391323] RDX: 00007f4c97fc8000 RSI: 000055ced3487b70 RDI: 0000000000000021 [921709.391325] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007ffdceb29fa8 [921709.391327] R10: 00007f4c97d2c848 R11: 0000000000000293 R12: 00007ffdceb29fa8 [921709.391329] R13: 00007ffdceb29fa0 R14: 000055ced34b0fa0 R15: 000055ced34bcf90 [921709.391343] INFO: task mysqld:1505 blocked for more than 241 seconds. [921709.391348] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391351] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391355] task:mysqld state:D stack: 0 pid: 1505 ppid: 1 flags:0x00000000 [921709.391359] Call Trace: [921709.391361] __schedule+0x44c/0x8a0 [921709.391366] schedule+0x4f/0xc0 [921709.391370] md_write_start+0x150/0x240 [921709.391373] ? wait_woken+0x80/0x80 [921709.391376] raid5_make_request+0x88/0x890 [raid456] [921709.391386] ? wait_woken+0x80/0x80 [921709.391388] ? mempool_kmalloc+0x17/0x20 [921709.391391] md_handle_request+0x12d/0x1a0 [921709.391395] ? __part_start_io_acct+0x51/0xf0 [921709.391398] md_submit_bio+0xca/0x100 [921709.391401] submit_bio_noacct+0x112/0x4f0 [921709.391404] submit_bio+0x51/0x1a0 [921709.391407] ? _cond_resched+0x19/0x30 [921709.391411] ? kmem_cache_alloc+0x38e/0x440 [921709.391414] ? ext4_init_io_end+0x1f/0x50 [921709.391418] ext4_io_submit+0x4d/0x60 [921709.391421] ext4_writepages+0x2c6/0xcd0 [921709.391423] ? __schedule+0x454/0x8a0 [921709.391427] ? hrtimer_start_range_ns+0x1aa/0x2f0 [921709.391430] ? timerqueue_del+0x24/0x50 [921709.391435] ? futex_wait+0x1ed/0x270 [921709.391439] do_writepages+0x43/0xd0 [921709.391443] ? do_writepages+0x43/0xd0 [921709.391445] ? futex_wake+0x7c/0x180 [921709.391449] __filemap_fdatawrite_range+0xcc/0x110 [921709.391453] file_write_and_wait_range+0x74/0xc0 [921709.391456] ext4_sync_file+0xf5/0x350 [921709.391459] vfs_fsync_range+0x49/0x80 [921709.391463] ? __fget_light+0x62/0x80 [921709.391466] do_fsync+0x3d/0x70 [921709.391469] __x64_sys_fsync+0x14/0x20 [921709.391472] do_syscall_64+0x38/0x90 [921709.391475] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391478] RIP: 0033:0x7f36188d48db [921709.391481] RSP: 002b:00007f35f05487d0 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [921709.391484] RAX: ffffffffffffffda RBX: 00007f360502bf50 RCX: 00007f36188d48db [921709.391486] RDX: 000d239200000000 RSI: 0000000000000000 RDI: 000000000000000a [921709.391488] RBP: 00007f35f0548a00 R08: 0000000000000000 R09: 00007f360502beb0 [921709.391490] R10: 00007f35f0548ccf R11: 0000000000000293 R12: 00007f36042e9610 [921709.391491] R13: 000000000000000a R14: 0000000000000b00 R15: 0000000000000000 [921709.391505] INFO: task mdcheck:40625 blocked for more than 241 seconds. [921709.391510] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391514] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391518] task:mdcheck state:D stack: 0 pid:40625 ppid: 1 flags:0x00000000 [921709.391522] Call Trace: [921709.391524] __schedule+0x44c/0x8a0 [921709.391528] schedule+0x4f/0xc0 [921709.391532] schedule_timeout+0x202/0x290 [921709.391535] ? do_truncate+0x78/0xd0 [921709.391538] ? try_to_wake_up+0x66/0x5c0 [921709.391542] wait_for_completion+0x94/0x100 [921709.391547] kthread_stop+0x71/0x170 [921709.391550] md_unregister_thread+0x47/0x80 [921709.391554] md_reap_sync_thread+0x24/0x200 [921709.391557] action_store+0x160/0x2e0 [921709.391561] md_attr_store+0x82/0xc0 [921709.391564] sysfs_kf_write+0x3e/0x50 [921709.391567] kernfs_fop_write_iter+0x138/0x1c0 [921709.391571] new_sync_write+0x117/0x1b0 [921709.391575] vfs_write+0x1ca/0x280 [921709.391578] ksys_write+0x67/0xe0 [921709.391581] __x64_sys_write+0x1a/0x20 [921709.391584] do_syscall_64+0x38/0x90 [921709.391588] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [921709.391591] RIP: 0033:0x7f8c6d2791e7 [921709.391593] RSP: 002b:00007ffea5d97178 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [921709.391597] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f8c6d2791e7 [921709.391599] RDX: 0000000000000005 RSI: 000055b618a3eb40 RDI: 0000000000000001 [921709.391600] RBP: 000055b618a3eb40 R08: 000000000000000a R09: 0000000000000004 [921709.391602] R10: 000055b616e97017 R11: 0000000000000246 R12: 0000000000000005 [921709.391604] R13: 00007f8c6d3546a0 R14: 00007f8c6d3554a0 R15: 00007f8c6d3548a0 [921709.391607] INFO: task md1_resync:40644 blocked for more than 241 seconds. [921709.391611] Not tainted 5.11.0-27-generic #29~20.04.1-Ubuntu [921709.391615] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [921709.391619] task:md1_resync state:D stack: 0 pid:40644 ppid: 2 flags:0x00004000 [921709.391623] Call Trace: [921709.391625] __schedule+0x44c/0x8a0 [921709.391631] schedule+0x4f/0xc0 [921709.391635] md_do_sync.cold+0x8b7/0x953 [921709.391639] ? wait_woken+0x80/0x80 [921709.391642] ? kernel_sigaction+0x3b/0xd0 [921709.391646] md_thread+0xb1/0x170 [921709.391650] kthread+0x114/0x150 [921709.391652] ? md_start_sync+0x60/0x60 [921709.391655] ? kthread_park+0x90/0x90 [921709.391658] ret_from_fork+0x22/0x30 Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sda3[0] sdd3[3] sdb3[1] sde3[5] sdc3[6] 15625582592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU] [================>....] check = 84.2% (3289834208/3906395648) finish=82779.6min speed=124K/sec md0 : active raid1 sda2[0] sdb2[1] sdc2[2](S) 488128 blocks super 1.2 [2/2] [UU] unused devices: