For a while now, I've been experiencing periodic problems where my RAID1 array would go into a rebuilding state and the system would hang with reports of a cpu soft lock. Indications where that the soft lock was due to KVM virts using the array while it was rebuilding. To work around the problem I've been monitoring /proc/mdstat and pausing the virts when the rebuild occurs. Recently I've tracked the rebuild down to checkarray's roughly monthly execution. As you can see from this log output, a few minutes after checkarray kicks off the problem occurs.
Aug 7 00:57:01 kvm-host kernel: [10854610.167521] md: data-check of RAID array md0
Aug 7 00:57:01 kvm-host kernel: [10854610.167531] md: minimum _guaranteed_ speed: 100000 KB/sec/disk.
Aug 7 00:57:01 kvm-host kernel: [10854610.167538] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Aug 7 00:57:01 kvm-host kernel: [10854610.167550] md: using 128k window, over a total of 240832 blocks.
Aug 7 00:57:01 kvm-host kernel: [10854610.181578] md: delaying data-check of md1 until md0 has finished (they share one or more physical units)
Aug 7 00:57:01 kvm-host kernel: [10854610.184873] md: delaying data-check of md2 until md1 has finished (they share one or more physical units)
Aug 7 00:57:01 kvm-host kernel: [10854610.184922] md: delaying data-check of md1 until md0 has finished (they share one or more physical units)
Aug 7 00:57:01 kvm-host kernel: [10854610.186091] md: delaying data-check of md3 until md0 has finished (they share one or more physical units)
Aug 7 00:57:01 kvm-host kernel: [10854610.186097] md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
Aug 7 00:57:01 kvm-host kernel: [10854610.186596] md: delaying data-check of md2 until md3 has finished (they share one or more physical units)
Aug 7 00:57:06 kvm-host kernel: [10854614.292430] md: md0: data-check done.
Aug 7 00:57:06 kvm-host kernel: [10854614.377928] md: delaying data-check of md2 until md3 has finished (they share one or more physical units)
Aug 7 00:57:06 kvm-host kernel: [10854614.377942] md: data-check of RAID array md3
Aug 7 00:57:06 kvm-host kernel: [10854614.377953] md: minimum _guaranteed_ speed: 100000 KB/sec/disk.
Aug 7 00:57:06 kvm-host kernel: [10854614.377961] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Aug 7 00:57:06 kvm-host kernel: [10854614.377976] md: using 128k window, over a total of 1943744384 blocks.
Aug 7 00:57:06 kvm-host kernel: [10854614.377984] md: delaying data-check of md1 until md3 has finished (they share one or more physical units)
Aug 7 01:03:06 kvm-host kernel: [10854974.265081] kvm D 00000000ffffffff 0 16486 1 0x00000000
Aug 7 01:03:06 kvm-host kernel: [10854974.265093] ffff8800987f9808 0000000000000082 0000000000015dc0 0000000000015dc0
Aug 7 01:03:06 kvm-host kernel: [10854974.265104] ffff8800090c9ad0 ffff8800987f9fd8 0000000000015dc0 ffff8800090c9700
Aug 7 01:03:06 kvm-host kernel: [10854974.265113] 0000000000015dc0 ffff8800987f9fd8 0000000000015dc0 ffff8800090c9ad0
Aug 7 01:03:06 kvm-host kernel: [10854974.265122] Call Trace:
Aug 7 01:03:06 kvm-host kernel: [10854974.265162] [<ffffffffa0052685>] wait_barrier+0xf5/0x140 [raid1]
Aug 7 01:03:06 kvm-host kernel: [10854974.265178] [<ffffffff8105a960>] ? default_wake_function+0x0/0x20
Aug 7 01:03:06 kvm-host kernel: [10854974.265192] [<ffffffffa0055651>] make_request+0x51/0x750 [raid1]
Aug 7 01:03:06 kvm-host kernel: [10854974.265204] [<ffffffff812049bb>] ? __ext4_handle_dirty_metadata+0x7b/0x100
Aug 7 01:03:06 kvm-host kernel: [10854974.265216] [<ffffffff8142a7a0>] md_make_request+0xc0/0x130
Aug 7 01:03:06 kvm-host kernel: [10854974.265224] [<ffffffff81438d8d>] ? _dm_request+0x10d/0x1e0
Aug 7 01:03:06 kvm-host kernel: [10854974.265233] [<ffffffff812a1411>] generic_make_request+0x1b1/0x4f0
Aug 7 01:03:06 kvm-host kernel: [10854974.265242] [<ffffffff81437ddb>] ? dm_get_table+0x4b/0x60
Aug 7 01:03:06 kvm-host kernel: [10854974.265250] [<ffffffff812a17d0>] submit_bio+0x80/0x110
Aug 7 01:03:06 kvm-host kernel: [10854974.265260] [<ffffffff810397f9>] ? default_spin_lock_flags+0x9/0x10
Aug 7 01:03:06 kvm-host kernel: [10854974.265268] [<ffffffff811778c1>] dio_bio_submit+0x61/0x90
Aug 7 01:03:06 kvm-host kernel: [10854974.265276] [<ffffffff81178430>] direct_io_worker+0x250/0x370
Aug 7 01:03:06 kvm-host kernel: [10854974.265284] [<ffffffff81178825>] __blockdev_direct_IO+0x205/0x480
Aug 7 01:03:06 kvm-host kernel: [10854974.265291] [<ffffffff811752b0>] ? blkdev_get_blocks+0x0/0xc0
Aug 7 01:03:06 kvm-host kernel: [10854974.265299] [<ffffffff810ff3c4>] ? generic_writepages+0x24/0x30
Aug 7 01:03:06 kvm-host kernel: [10854974.265308] [<ffffffff810f6400>] ? __filemap_fdatawrite_range+0x40/0x60
Aug 7 01:03:06 kvm-host kernel: [10854974.265315] [<ffffffff8117623e>] blkdev_direct_IO+0x4e/0x50
Aug 7 01:03:06 kvm-host kernel: [10854974.265322] [<ffffffff811752b0>] ? blkdev_get_blocks+0x0/0xc0
Aug 7 01:03:06 kvm-host kernel: [10854974.265330] [<ffffffff810f6642>] generic_file_direct_write+0xc2/0x190
Aug 7 01:03:06 kvm-host kernel: [10854974.265338] [<ffffffff810f7c15>] __generic_file_aio_write+0x335/0x470
Aug 7 01:03:06 kvm-host kernel: [10854974.265347] [<ffffffff8105a960>] ? default_wake_function+0x0/0x20
Aug 7 01:03:06 kvm-host kernel: [10854974.265355] [<ffffffff8117581c>] blkdev_aio_write+0x3c/0xa0
Aug 7 01:03:06 kvm-host kernel: [10854974.265363] [<ffffffff81145faa>] do_sync_write+0xfa/0x140
Aug 7 01:03:06 kvm-host kernel: [10854974.265371] [<ffffffff81053a70>] ? __dequeue_entity+0x30/0x50
Aug 7 01:03:06 kvm-host kernel: [10854974.265380] [<ffffffff81085b40>] ? autoremove_wake_function+0x0/0x40
Aug 7 01:03:06 kvm-host kernel: [10854974.265388] [<ffffffff81145eb0>] ? do_sync_write+0x0/0x140
Aug 7 01:03:06 kvm-host kernel: [10854974.265396] [<ffffffff81145ebb>] ? do_sync_write+0xb/0x140
Aug 7 01:03:06 kvm-host kernel: [10854974.265404] [<ffffffff811462a8>] vfs_write+0xb8/0x1a0
Aug 7 01:03:06 kvm-host kernel: [10854974.265412] [<ffffffff81146c42>] sys_pwrite64+0x82/0xa0
Aug 7 01:03:06 kvm-host kernel: [10854974.265422] [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: mdadm 2.6.7.1-1ubuntu15
ProcVersionSignature: Ubuntu 2.6.32-33.70-server 2.6.32.41+drm33.18
Uname: Linux 2.6.32-33-server x86_64
Architecture: amd64
Date: Mon Aug 8 23:12:46 2011
MDadmExamine.dev.sda: Error: command ['/sbin/mdadm', '-E', '/dev/sda'] failed with exit code 1: mdadm: No md superblock detected on /dev/sda.
MDadmExamine.dev.sda2: Error: command ['/sbin/mdadm', '-E', '/dev/sda2'] failed with exit code 1: mdadm: No md superblock detected on /dev/sda2.
MDadmExamine.dev.sda5: Error: command ['/sbin/mdadm', '-E', '/dev/sda5'] failed with exit code 1: mdadm: No md superblock detected on /dev/sda5.
MDadmExamine.dev.sdb: Error: command ['/sbin/mdadm', '-E', '/dev/sdb'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdb.
MDadmExamine.dev.sdb2: Error: command ['/sbin/mdadm', '-E', '/dev/sdb2'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdb2.
MDadmExamine.dev.sdb5: Error: command ['/sbin/mdadm', '-E', '/dev/sdb5'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdb5.
MDadmExamine.dev.sdc: Error: command ['/sbin/mdadm', '-E', '/dev/sdc'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdc.
MDadmExamine.dev.sdc2: Error: command ['/sbin/mdadm', '-E', '/dev/sdc2'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdc2.
MDadmExamine.dev.sdc5: Error: command ['/sbin/mdadm', '-E', '/dev/sdc5'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdc5.
MDadmExamine.dev.sdd: Error: command ['/sbin/mdadm', '-E', '/dev/sdd'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdd.
MDadmExamine.dev.sdd1: Error: command ['/sbin/mdadm', '-E', '/dev/sdd1'] failed with exit code 1: mdadm: No md superblock detected on /dev/sdd1.
MDadmExamine.dev.sde: Error: command ['/sbin/mdadm', '-E', '/dev/sde'] failed with exit code 1: mdadm: No md superblock detected on /dev/sde.
MDadmExamine.dev.sde1: Error: command ['/sbin/mdadm', '-E', '/dev/sde1'] failed with exit code 1: mdadm: No md superblock detected on /dev/sde1.
MachineType: Gigabyte Technology Co., Ltd. GA-MA785GM-US2H
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.32-33-server root=UUID=03cdd2fa-216f-451b-a06c-60e7ccd69238 ro quiet splash
ProcEnviron:
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: mdadm
dmi.bios.date: 09/16/2009
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: F5
dmi.board.name: GA-MA785GM-US2H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrF5:bd09/16/2009:svnGigabyteTechnologyCo.,Ltd.:pnGA-MA785GM-US2H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-MA785GM-US2H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-MA785GM-US2H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.
Related reports:
http://
https:/
https:/