Pretty fed up with people saying this could be so many different issues. So much so that i finally decided to risk my data to prove it.... read the following. ***___This has got to be the card / chipset / sata_mv driver._____*** Short and simple version of my issues: - This does not depend on drive types - Appears to be caused by MV88SX6081 chipset - Could be a problem in SATA_MV driver - I need replacement controller suggestions Details to all non believers (it’s not a power / hardware issue): I moved 5 of the 7 drives to my onboard controller (have 6 sata ports on the mobo, last was used by the system drive). Left 2 of the western digital drives on the MV88SX6081 8-port SATA II: - sdg - sdh After the advice of some through email, I unplugged everything that wasn't needed. They assumed that it could have been power giving the number of drives I had in the machine. What was left on a tx750w corsair power supply: - mobo (c2d, 4gb ram) - 7 sata raid drives - spread across multiple power supply rails - 1 sata system drive - Super Micro SAT2-MV8 (MV88SX6081 8-port SATA II) - intel pcie 10/100/1000 network card Then I replaced the sate cables 1 more time with old cables I knew worked. I also threw in the brand new controller card as well (have a few spares lying around). I brought everything up and upgraded to: Then I started to rebuild the raid. Everything went fine, no freezes. **This was the first indication that this only happens under heavy load on multiple ports as has been brought up before. So then I started copying data over. About 180GB's the card hard reset both of the drives attached to it and knocked them both out of the raid. **This was also significantly different from before when I was utilizing all the ports as it seemed to work great for quite some time, it wasn't until I was well into the process that the card finally gave up. See the attached dmesg and /var/log/messages. This is the 2nd time I’ve had this card degrade my raid and almost give me a heart attack. The cards are going in the trash at this point. I'm open to suggestions as to possibly replacement. I don’t need a hardware raid card, just a decent controller with great *nix support and lots of ports. ::sigh:: I don’t know who to contact but this is the end of the line for me with this controller and hopefully my issues. Attempting to get my data back as we speak with 2 failed drives in a raid 5... wonderful times. dmsg of the event: [ 1061.040118] md: recovery of RAID array md1 [ 1061.040120] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [ 1061.040122] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery. [ 1061.040126] md: using 128k window, over a total of 488383744 blocks. [11208.852220] md: md1: recovery done. [11209.020072] RAID5 conf printout: [11209.020076] --- rd:7 wd:7 [11209.020079] disk 0, o:1, dev:sdd1 [11209.020080] disk 1, o:1, dev:sdb1 [11209.020081] disk 2, o:1, dev:sdh1 [11209.020082] disk 3, o:1, dev:sdc1 [11209.020083] disk 4, o:1, dev:sdf1 [11209.020084] disk 5, o:1, dev:sde1 [11209.020085] disk 6, o:1, dev:sdg1 [19844.431690] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled [19844.433148] SGI XFS Quota Management subsystem [19844.442507] Filesystem "md1": Disabling barriers, trial barrier write failed [19844.442658] XFS mounting filesystem md1 [19844.893398] Ending clean XFS mount for filesystem: md1 [27027.170016] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [27027.170041] ata5.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 [27027.170041] res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) [27027.170083] ata5.00: status: { DRDY } [27027.170099] ata5: hard resetting link [27027.680034] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [27027.720050] ata5.00: max_sectors limited to 256 for NCQ [27027.780047] ata5.00: max_sectors limited to 256 for NCQ [27027.780050] ata5.00: configured for UDMA/133 [27027.780055] end_request: I/O error, dev sdg, sector 73 [27027.780073] md: super_written gets error=-5, uptodate=0 [27027.780076] raid5: Disk failure on sdg1, disabling device. [27027.780077] raid5: Operation continuing on 6 devices. [27027.780117] ata5: EH complete [27027.780674] sd 4:0:0:0: [sdg] 976773168 512-byte hardware sectors (500108 MB) [27027.780800] sd 4:0:0:0: [sdg] Write Protect is off [27027.780803] sd 4:0:0:0: [sdg] Mode Sense: 00 3a 00 00 [27027.781038] sd 4:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [27057.930015] ata12.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen [27057.930039] ata12.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0 [27057.930040] res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) [27057.930081] ata12.00: status: { DRDY } [27057.930098] ata12: hard resetting link [27058.440033] ata12: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [27058.480049] ata12.00: max_sectors limited to 256 for NCQ [27058.540047] ata12.00: max_sectors limited to 256 for NCQ [27058.540050] ata12.00: configured for UDMA/133 [27058.540055] end_request: I/O error, dev sdh, sector 71 [27058.540072] md: super_written gets error=-5, uptodate=0 [27058.540075] raid5: Disk failure on sdh1, disabling device. [27058.540076] raid5: Operation continuing on 5 devices. [27058.540113] ata12: EH complete [27058.540754] sd 11:0:0:0: [sdh] 976773168 512-byte hardware sectors (500108 MB) [27058.540879] sd 11:0:0:0: [sdh] Write Protect is off [27058.540882] sd 11:0:0:0: [sdh] Mode Sense: 00 3a 00 00 [27058.541070] sd 11:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [27058.584017] RAID5 conf printout: [27058.584020] --- rd:7 wd:5 [27058.584022] disk 0, o:1, dev:sdd1 [27058.584023] disk 1, o:1, dev:sdb1 [27058.584024] disk 2, o:0, dev:sdh1 [27058.584025] disk 3, o:1, dev:sdc1 [27058.584027] disk 4, o:1, dev:sdf1 [27058.584028] disk 5, o:1, dev:sde1 [27058.584029] disk 6, o:0, dev:sdg1 [27061.521245] BUG: soft lockup - CPU#1 stuck for 61s! [smbd:28171] [27061.521251] Modules linked in: xfs aes_x86_64 aes_generic ecb crypto_blkcipher ecryptfs ipv6 af_packet iptable_filter ip_tables x_tables ac sbp2 parport_pc lp parport loop psmouse pcspkr serio_raw iTCO_wdt iTCO_vendor_support evdev button intel_agp snd_hda_intel snd_pcm shpchp snd_timer pci_hotplug snd soundcore snd_page_alloc ext3 jbd mbcache sd_mod crc_t10dif sg pata_acpi pata_marvell usbhid hid ohci1394 ieee1394 sata_mv ata_generic ata_piix libata scsi_mod dock sky2 e1000e ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_log dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [27061.521251] CPU 1: [27061.521251] Modules linked in: xfs aes_x86_64 aes_generic ecb crypto_blkcipher ecryptfs ipv6 af_packet iptable_filter ip_tables x_tables ac sbp2 parport_pc lp parport loop psmouse pcspkr serio_raw iTCO_wdt iTCO_vendor_support evdev button intel_agp snd_hda_intel snd_pcm shpchp snd_timer pci_hotplug snd soundcore snd_page_alloc ext3 jbd mbcache sd_mod crc_t10dif sg pata_acpi pata_marvell usbhid hid ohci1394 ieee1394 sata_mv ata_generic ata_piix libata scsi_mod dock sky2 e1000e ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_log dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [27061.521251] Pid: 28171, comm: smbd Not tainted 2.6.27-9-server #1 [27061.521251] RIP: 0010:[] [] find_get_pages+0x6c/0x110 [27061.521251] RSP: 0018:ffff880129453358 EFLAGS: 00000246 [27061.521251] RAX: ffff880128d89330 RBX: ffff880129453398 RCX: 0000000000000002 [27061.521251] RDX: 0000000000000003 RSI: 0000000000000000 RDI: ffffe200022e9e80 [27061.521251] RBP: ffff880129453308 R08: ffffe200009df6c8 R09: 0000000000000005 [27061.521251] R10: 0000000000000037 R11: 00000000001c5778 R12: ffffffff802b6b29 [27061.521251] R13: ffff880123a107d0 R14: ffffe20001c6f6c0 R15: 0000000000000286 [27061.521251] FS: 00007fb72cdf6700(0000) GS:ffff88012fc02980(0000) knlGS:0000000000000000 [27061.521251] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [27061.521251] CR2: 00007f1648629000 CR3: 000000012956d000 CR4: 00000000000006e0 [27061.521251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [27061.521251] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [27061.521251] [27061.521251] Call Trace: [27061.521251] [] ? find_get_pages+0x43/0x110 [27061.521251] [] ? pagevec_lookup+0x24/0x30 [27061.521251] [] ? xfs_cluster_write+0xad/0x180 [xfs] [27061.521251] [] ? xfs_page_state_convert+0x498/0x760 [xfs] [27061.521251] [] ? xfs_vm_writepage+0x71/0x120 [xfs] [27061.521251] [] ? pageout+0x124/0x270 [27061.521251] [] ? page_waitqueue+0xa/0x90 [27061.521251] [] ? shrink_page_list+0x34d/0x530 [27061.521251] [] ? __isolate_lru_page+0x79/0xb0 [27061.521251] [] ? isolate_lru_pages+0x8a/0x220 [27061.521251] [] ? shrink_inactive_list+0x1a2/0x4b0 [27061.521251] [] ? shrink_zone+0x7b/0x160 [27061.521251] [] ? shrink_zones+0x8d/0x150 [27061.521251] [] ? do_try_to_free_pages+0x86/0x2e0 [27061.521251] [] ? try_to_free_pages+0x67/0x70 [27061.521251] [] ? isolate_pages_global+0x0/0x50 [27061.521251] [] ? __alloc_pages_internal+0x241/0x510 [27061.521251] [] ? alloc_pages_current+0xad/0x110 [27061.521251] [] ? __page_cache_alloc+0x67/0x80 [27061.521251] [] ? __grab_cache_page+0x63/0xb0 [27061.521251] [] ? block_write_begin+0x89/0xf0 [27061.521251] [] ? xfs_vm_write_begin+0x2a/0x30 [xfs] [27061.521251] [] ? xfs_get_blocks+0x0/0x20 [xfs] [27061.521251] [] ? generic_perform_write+0xbc/0x1c0 [27061.521251] [] ? generic_file_buffered_write+0x92/0x170 [27061.521251] [] ? xfs_write+0x6b3/0x9b0 [xfs] [27061.521251] [] ? apparmor_socket_recvmsg+0x19/0x20 [27061.521251] [] ? memset_c+0x20/0x30 [27061.521251] [] ? xfs_file_aio_write+0x58/0x60 [xfs] [27061.521251] [] ? do_sync_write+0xf9/0x140 [27061.521251] [] ? do_sync_read+0xf9/0x140 [27061.521251] [] ? autoremove_wake_function+0x0/0x40 [27061.521251] [] ? aa_file_permission+0x21/0xf0 [27061.521251] [] ? apparmor_file_permission+0x28/0x30 [27061.521251] [] ? security_file_permission+0x16/0x20 [27061.521251] [] ? vfs_write+0xcb/0x130 [27061.521251] [] ? sys_pwrite64+0x9a/0xa0 [27061.521251] [] ? system_call_fastpath+0x16/0x1b [27061.521251] [27095.080066] RAID5 conf printout: [27095.080071] --- rd:7 wd:5 [27095.080074] disk 0, o:1, dev:sdd1 [27095.080076] disk 1, o:1, dev:sdb1 [27095.080077] disk 2, o:0, dev:sdh1 [27095.080079] disk 3, o:1, dev:sdc1 [27095.080080] disk 4, o:1, dev:sdf1 [27095.080082] disk 5, o:1, dev:sde1 [27095.080090] RAID5 conf printout: [27095.080091] --- rd:7 wd:5 [27095.080092] disk 0, o:1, dev:sdd1 [27095.080093] disk 1, o:1, dev:sdb1 [27095.080094] disk 2, o:0, dev:sdh1 [27095.080095] disk 3, o:1, dev:sdc1 [27095.080097] disk 4, o:1, dev:sdf1 [27095.080098] disk 5, o:1, dev:sde1 [27095.140011] RAID5 conf printout: [27095.140017] --- rd:7 wd:5 [27095.140019] disk 0, o:1, dev:sdd1 [27095.140022] disk 1, o:1, dev:sdb1 [27095.140024] disk 3, o:1, dev:sdc1 [27095.140026] disk 4, o:1, dev:sdf1 [27095.140027] disk 5, o:1, dev:sde1 [27095.140511] Buffer I/O error on device md1, logical block 455870845 [27095.140545] lost page write due to I/O error on md1 [27095.140550] Buffer I/O error on device md1, logical block 455870846 [27095.140567] lost page write due to I/O error on md1 [27095.140569] Buffer I/O error on device md1, logical block 455870847 [27095.140585] lost page write due to I/O error on md1 [27095.140587] Buffer I/O error on device md1, logical block 455870848 [27095.140604] lost page write due to I/O error on md1 [27095.140606] Buffer I/O error on device md1, logical block 455870849 [27095.140622] lost page write due to I/O error on md1 [27095.140624] Buffer I/O error on device md1, logical block 455870850 [27095.140641] lost page write due to I/O error on md1 [27095.140642] Buffer I/O error on device md1, logical block 455870851 [27095.140659] lost page write due to I/O error on md1 [27095.140661] Buffer I/O error on device md1, logical block 455870852 [27095.140677] lost page write due to I/O error on md1 [27095.140679] Buffer I/O error on device md1, logical block 455870853 [27095.140696] lost page write due to I/O error on md1 [27095.140697] Buffer I/O error on device md1, logical block 455870854 [27095.140714] lost page write due to I/O error on md1 [27095.141327] I/O error in filesystem ("md1") meta-data dev md1 block 0xaeaa9810 ("xlog_iodone") error 5 buf count 12288 [27095.141359] xfs_force_shutdown(md1,0x2) called from line 1056 of file /build/buildd/linux-2.6.27/fs/xfs/xfs_log.c. Return address = 0xffffffffa04c80d3 [27095.141380] Filesystem "md1": Log I/O Error Detected. Shutting down filesystem: md1 [27095.141407] Please umount the filesystem, and rectify the problem(s) [27100.140015] Filesystem "md1": xfs_log_force: error 5 returned. [27113.440011] Filesystem "md1": xfs_log_force: error 5 returned. [27143.440010] Filesystem "md1": xfs_log_force: error 5 returned. [27173.440009] Filesystem "md1": xfs_log_force: error 5 returned. [27203.440012] Filesystem "md1": xfs_log_force: error 5 returned. /var/log/messages: Nov 30 18:39:24 isis kernel: [ 1061.040118] md: recovery of RAID array md1 Nov 30 18:39:24 isis kernel: [ 1061.040120] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Nov 30 18:39:24 isis kernel: [ 1061.040122] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery. Nov 30 18:39:24 isis kernel: [ 1061.040126] md: using 128k window, over a total of 488383744 blocks. Nov 30 19:02:08 isis -- MARK -- Nov 30 19:22:08 isis -- MARK -- Nov 30 19:42:08 isis -- MARK -- Nov 30 20:02:08 isis -- MARK -- Nov 30 20:22:08 isis -- MARK -- Nov 30 20:42:08 isis -- MARK -- Nov 30 21:02:08 isis -- MARK -- Nov 30 21:22:08 isis -- MARK -- Nov 30 21:28:32 isis kernel: [11208.852220] md: md1: recovery done. Nov 30 21:28:32 isis kernel: [11209.020072] RAID5 conf printout: Nov 30 21:28:32 isis kernel: [11209.020076] --- rd:7 wd:7 Nov 30 21:28:32 isis kernel: [11209.020079] disk 0, o:1, dev:sdd1 Nov 30 21:28:32 isis kernel: [11209.020080] disk 1, o:1, dev:sdb1 Nov 30 21:28:32 isis kernel: [11209.020081] disk 2, o:1, dev:sdh1 Nov 30 21:28:32 isis kernel: [11209.020082] disk 3, o:1, dev:sdc1 Nov 30 21:28:32 isis kernel: [11209.020083] disk 4, o:1, dev:sdf1 Nov 30 21:28:32 isis kernel: [11209.020084] disk 5, o:1, dev:sde1 Nov 30 21:28:32 isis kernel: [11209.020085] disk 6, o:1, dev:sdg1 Nov 30 21:42:08 isis -- MARK -- Nov 30 22:02:08 isis -- MARK -- Nov 30 22:22:08 isis -- MARK -- Nov 30 22:42:08 isis -- MARK -- Nov 30 23:02:08 isis -- MARK -- Nov 30 23:22:08 isis -- MARK -- Nov 30 23:42:08 isis -- MARK -- Nov 30 23:52:27 isis kernel: [19844.431690] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled Nov 30 23:52:27 isis kernel: [19844.433148] SGI XFS Quota Management subsystem Nov 30 23:52:27 isis kernel: [19844.442507] Filesystem "md1": Disabling barriers, trial barrier write failed Nov 30 23:52:27 isis kernel: [19844.442658] XFS mounting filesystem md1 Dec 1 00:22:08 isis -- MARK -- Dec 1 00:42:08 isis -- MARK -- Dec 1 01:02:08 isis -- MARK -- Dec 1 01:22:08 isis -- MARK -- Dec 1 01:42:08 isis -- MARK -- Dec 1 01:52:10 isis kernel: [27027.170099] ata5: hard resetting link Dec 1 01:52:10 isis kernel: [27027.680034] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Dec 1 01:52:11 isis kernel: [27027.720050] ata5.00: max_sectors limited to 256 for NCQ Dec 1 01:52:11 isis kernel: [27027.780047] ata5.00: max_sectors limited to 256 for NCQ Dec 1 01:52:11 isis kernel: [27027.780050] ata5.00: configured for UDMA/133 Dec 1 01:52:11 isis kernel: [27027.780073] md: super_written gets error=-5, uptodate=0 Dec 1 01:52:11 isis kernel: [27027.780117] ata5: EH complete Dec 1 01:52:11 isis kernel: [27027.780674] sd 4:0:0:0: [sdg] 976773168 512-byte hardware sectors (500108 MB) Dec 1 01:52:11 isis kernel: [27027.780800] sd 4:0:0:0: [sdg] Write Protect is off Dec 1 01:52:11 isis kernel: [27027.781038] sd 4:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Dec 1 01:52:41 isis kernel: [27057.930098] ata12: hard resetting link Dec 1 01:52:41 isis kernel: [27058.440033] ata12: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Dec 1 01:52:41 isis kernel: [27058.480049] ata12.00: max_sectors limited to 256 for NCQ Dec 1 01:52:41 isis kernel: [27058.540047] ata12.00: max_sectors limited to 256 for NCQ Dec 1 01:52:41 isis kernel: [27058.540050] ata12.00: configured for UDMA/133 Dec 1 01:52:41 isis kernel: [27058.540072] md: super_written gets error=-5, uptodate=0 Dec 1 01:52:41 isis kernel: [27058.540113] ata12: EH complete Dec 1 01:52:41 isis kernel: [27058.540754] sd 11:0:0:0: [sdh] 976773168 512-byte hardware sectors (500108 MB) Dec 1 01:52:41 isis kernel: [27058.540879] sd 11:0:0:0: [sdh] Write Protect is off Dec 1 01:52:41 isis kernel: [27058.541070] sd 11:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Dec 1 01:52:41 isis kernel: [27058.584017] RAID5 conf printout: Dec 1 01:52:41 isis kernel: [27058.584020] --- rd:7 wd:5 Dec 1 01:52:41 isis kernel: [27058.584022] disk 0, o:1, dev:sdd1 Dec 1 01:52:41 isis kernel: [27058.584023] disk 1, o:1, dev:sdb1 Dec 1 01:52:41 isis kernel: [27058.584024] disk 2, o:0, dev:sdh1 Dec 1 01:52:41 isis kernel: [27058.584025] disk 3, o:1, dev:sdc1 Dec 1 01:52:41 isis kernel: [27058.584027] disk 4, o:1, dev:sdf1 Dec 1 01:52:41 isis kernel: [27058.584028] disk 5, o:1, dev:sde1 Dec 1 01:52:41 isis kernel: [27058.584029] disk 6, o:0, dev:sdg1 Dec 1 01:52:44 isis kernel: [27061.521251] Modules linked in: xfs aes_x86_64 aes_generic ecb crypto_blkcipher ecryptfs ipv6 af_packet iptable_filter ip_tables x_tables ac sbp2 parport_pc lp parport loop psmouse pcspkr serio_raw iTCO_wdt iTCO_vendor_support evdev button intel_agp snd_hda_intel snd_pcm shpchp snd_timer pci_hotplug snd soundcore snd_page_alloc ext3 jbd mbcache sd_mod crc_t10dif sg pata_acpi pata_marvell usbhid hid ohci1394 ieee1394 sata_mv ata_generic ata_piix libata scsi_mod dock sky2 e1000e ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_log dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse Dec 1 01:52:44 isis kernel: [27061.521251] CPU 1: Dec 1 01:52:44 isis kernel: [27061.521251] Modules linked in: xfs aes_x86_64 aes_generic ecb crypto_blkcipher ecryptfs ipv6 af_packet iptable_filter ip_tables x_tables ac sbp2 parport_pc lp parport loop psmouse pcspkr serio_raw iTCO_wdt iTCO_vendor_support evdev button intel_agp snd_hda_intel snd_pcm shpchp snd_timer pci_hotplug snd soundcore snd_page_alloc ext3 jbd mbcache sd_mod crc_t10dif sg pata_acpi pata_marvell usbhid hid ohci1394 ieee1394 sata_mv ata_generic ata_piix libata scsi_mod dock sky2 e1000e ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod dm_mirror dm_log dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse Dec 1 01:52:44 isis kernel: [27061.521251] Pid: 28171, comm: smbd Not tainted 2.6.27-9-server #1 Dec 1 01:52:44 isis kernel: [27061.521251] RIP: 0010:[] [] find_get_pages+0x6c/0x110 Dec 1 01:52:44 isis kernel: [27061.521251] RSP: 0018:ffff880129453358 EFLAGS: 00000246 Dec 1 01:52:44 isis kernel: [27061.521251] RAX: ffff880128d89330 RBX: ffff880129453398 RCX: 0000000000000002 Dec 1 01:52:44 isis kernel: [27061.521251] RDX: 0000000000000003 RSI: 0000000000000000 RDI: ffffe200022e9e80 Dec 1 01:52:44 isis kernel: [27061.521251] RBP: ffff880129453308 R08: ffffe200009df6c8 R09: 0000000000000005 Dec 1 01:52:44 isis kernel: [27061.521251] R10: 0000000000000037 R11: 00000000001c5778 R12: ffffffff802b6b29 Dec 1 01:52:44 isis kernel: [27061.521251] R13: ffff880123a107d0 R14: ffffe20001c6f6c0 R15: 0000000000000286 Dec 1 01:52:44 isis kernel: [27061.521251] FS: 00007fb72cdf6700(0000) GS:ffff88012fc02980(0000) knlGS:0000000000000000 Dec 1 01:52:44 isis kernel: [27061.521251] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Dec 1 01:52:44 isis kernel: [27061.521251] CR2: 00007f1648629000 CR3: 000000012956d000 CR4: 00000000000006e0 Dec 1 01:52:44 isis kernel: [27061.521251] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Dec 1 01:52:44 isis kernel: [27061.521251] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Dec 1 01:52:44 isis kernel: [27061.521251] Dec 1 01:52:44 isis kernel: [27061.521251] Call Trace: Dec 1 01:52:44 isis kernel: [27061.521251] [] ? find_get_pages+0x43/0x110 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? pagevec_lookup+0x24/0x30 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_cluster_write+0xad/0x180 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_page_state_convert+0x498/0x760 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_vm_writepage+0x71/0x120 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? pageout+0x124/0x270 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? page_waitqueue+0xa/0x90 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? shrink_page_list+0x34d/0x530 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? __isolate_lru_page+0x79/0xb0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? isolate_lru_pages+0x8a/0x220 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? shrink_inactive_list+0x1a2/0x4b0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? shrink_zone+0x7b/0x160 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? shrink_zones+0x8d/0x150 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? do_try_to_free_pages+0x86/0x2e0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? try_to_free_pages+0x67/0x70 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? isolate_pages_global+0x0/0x50 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? __alloc_pages_internal+0x241/0x510 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? alloc_pages_current+0xad/0x110 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? __page_cache_alloc+0x67/0x80 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? __grab_cache_page+0x63/0xb0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? block_write_begin+0x89/0xf0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_vm_write_begin+0x2a/0x30 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_get_blocks+0x0/0x20 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? generic_perform_write+0xbc/0x1c0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? generic_file_buffered_write+0x92/0x170 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_write+0x6b3/0x9b0 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? apparmor_socket_recvmsg+0x19/0x20 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? memset_c+0x20/0x30 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? xfs_file_aio_write+0x58/0x60 [xfs] Dec 1 01:52:44 isis kernel: [27061.521251] [] ? do_sync_write+0xf9/0x140 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? do_sync_read+0xf9/0x140 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? autoremove_wake_function+0x0/0x40 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? aa_file_permission+0x21/0xf0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? apparmor_file_permission+0x28/0x30 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? security_file_permission+0x16/0x20 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? vfs_write+0xcb/0x130 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? sys_pwrite64+0x9a/0xa0 Dec 1 01:52:44 isis kernel: [27061.521251] [] ? system_call_fastpath+0x16/0x1b Dec 1 01:52:44 isis kernel: [27061.521251] Dec 1 01:53:18 isis kernel: [27095.080066] RAID5 conf printout: Dec 1 01:53:18 isis kernel: [27095.080071] --- rd:7 wd:5 Dec 1 01:53:18 isis kernel: [27095.080074] disk 0, o:1, dev:sdd1 Dec 1 01:53:18 isis kernel: [27095.080076] disk 1, o:1, dev:sdb1 Dec 1 01:53:18 isis kernel: [27095.080077] disk 2, o:0, dev:sdh1 Dec 1 01:53:18 isis kernel: [27095.080079] disk 3, o:1, dev:sdc1 Dec 1 01:53:18 isis kernel: [27095.080080] disk 4, o:1, dev:sdf1 Dec 1 01:53:18 isis kernel: [27095.080082] disk 5, o:1, dev:sde1 Dec 1 01:53:18 isis kernel: [27095.080090] RAID5 conf printout: Dec 1 01:53:18 isis kernel: [27095.080091] --- rd:7 wd:5 Dec 1 01:53:18 isis kernel: [27095.080092] disk 0, o:1, dev:sdd1 Dec 1 01:53:18 isis kernel: [27095.080093] disk 1, o:1, dev:sdb1 Dec 1 01:53:18 isis kernel: [27095.080094] disk 2, o:0, dev:sdh1 Dec 1 01:53:18 isis kernel: [27095.080095] disk 3, o:1, dev:sdc1 Dec 1 01:53:18 isis kernel: [27095.080097] disk 4, o:1, dev:sdf1 Dec 1 01:53:18 isis kernel: [27095.080098] disk 5, o:1, dev:sde1 Dec 1 01:53:18 isis kernel: [27095.140011] RAID5 conf printout: Dec 1 01:53:18 isis kernel: [27095.140017] --- rd:7 wd:5 Dec 1 01:53:18 isis kernel: [27095.140019] disk 0, o:1, dev:sdd1 Dec 1 01:53:18 isis kernel: [27095.140022] disk 1, o:1, dev:sdb1 Dec 1 01:53:18 isis kernel: [27095.140024] disk 3, o:1, dev:sdc1 Dec 1 01:53:18 isis kernel: [27095.140026] disk 4, o:1, dev:sdf1 Dec 1 01:53:18 isis kernel: [27095.140027] disk 5, o:1, dev:sde1 Dec 1 01:53:18 isis kernel: [27095.140545] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140567] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140585] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140604] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140622] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140641] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140659] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140677] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140696] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.140714] lost page write due to I/O error on md1 Dec 1 01:53:18 isis kernel: [27095.141359] xfs_force_shutdown(md1,0x2) called from line 1056 of file /build/buildd/linux-2.6.27/fs/xfs/xfs_log.c. Return address = 0xffffffffa04c80d3 Dec 1 01:53:23 isis kernel: [27100.140015] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:53:36 isis kernel: [27113.440011] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:54:06 isis kernel: [27143.440010] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:54:36 isis kernel: [27173.440009] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:55:06 isis kernel: [27203.440012] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:55:36 isis kernel: [27233.440011] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:56:06 isis kernel: [27263.440011] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:56:36 isis kernel: [27293.440010] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:57:06 isis kernel: [27323.440016] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:57:36 isis kernel: [27353.440015] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:58:06 isis kernel: [27383.440015] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 01:58:36 isis kernel: [27413.440016] Filesystem "md1": xfs_log_force: error 5 returned. ^^^^^^continues this for a while Dec 1 02:12:06 isis kernel: [28223.440015] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:12:36 isis kernel: [28253.440013] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:13:06 isis kernel: [28283.440014] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:13:36 isis kernel: [28313.440013] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:06 isis kernel: [28343.440013] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:36 isis kernel: [28373.440012] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28395.820448] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28395.820456] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28395.820462] xfs_force_shutdown(md1,0x1) called from line 420 of file /build/buildd/linux-2.6.27/fs/xfs/xfs_rw.c. Return address = 0xffffffffa04decc3 Dec 1 02:14:59 isis kernel: [28395.820466] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28395.820468] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28395.820471] xfs_force_shutdown(md1,0x1) called from line 420 of file /build/buildd/linux-2.6.27/fs/xfs/xfs_rw.c. Return address = 0xffffffffa04decc3 Dec 1 02:14:59 isis kernel: [28396.669470] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28396.669487] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28396.669517] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28396.669525] Filesystem "md1": xfs_log_force: error 5 returned. Dec 1 02:14:59 isis kernel: [28396.669635] Filesystem "md1": xfs_log_force: error 5 returned.