Not quite sure if it's the same bug, but maybe this adds some more data points to the set. AMI ID is ami-405c6934, which is the eu-west-1 EBS-backed variant of the image in question. Just a plain instance boot, connecting four EBS volumes, bundling them together as /dev/md0, putting XFS on top and running a benchmark as described in http://www.mysqlperformanceblog.com/2009/08/06/ec2ebs-single-and-raid-volumes-io-bencmark/. When doing the same (on the same volumes) from a 32bit AMI on a .small instance the problem does not occur (at least not over a few days), whereas the 64bit AMI crashed within hours. Dec 5 22:00:55 ip-10-234-243-114 kernel: [ 6.642845] JBD: barrier-based sync failed on sda1-8 - disabling barriers Dec 5 22:01:02 ip-10-234-243-114 kernel: [ 13.890171] eth0: no IPv6 routers present Dec 5 22:04:14 ip-10-234-243-114 kernel: [ 205.786453] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled Dec 5 22:04:14 ip-10-234-243-114 kernel: [ 205.789239] SGI XFS Quota Management subsystem Dec 5 22:04:14 ip-10-234-243-114 kernel: [ 205.790489] Filesystem "md0": Disabling barriers, trial barrier write failed Dec 5 22:04:14 ip-10-234-243-114 kernel: [ 205.807777] XFS mounting filesystem md0 Dec 5 22:04:15 ip-10-234-243-114 kernel: [ 206.368403] Ending clean XFS mount for filesystem: md0 Dec 5 22:07:36 ip-10-234-243-114 kernel: [ 407.339573] Filesystem "md0": Disabling barriers, trial barrier write failed Dec 5 22:07:36 ip-10-234-243-114 kernel: [ 407.340479] XFS mounting filesystem md0 Dec 5 22:07:36 ip-10-234-243-114 kernel: [ 407.593179] Ending clean XFS mount for filesystem: md0 Dec 5 22:17:01 ip-10-234-243-114 CRON[1222]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Dec 5 22:31:54 ip-10-234-243-114 kernel: [ 1865.812490] XFS mounting filesystem md0 Dec 5 22:31:54 ip-10-234-243-114 kernel: [ 1866.061941] Ending clean XFS mount for filesystem: md0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190047] INFO: task flush-9:0:1272 blocked for more than 120 seconds. Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190061] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190068] flush-9:0 D ffff880003e7d980 0 1272 2 0x00000000 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190072] ffff88014d79b640 0000000000000246 ffff880100000000 0000000000015980 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190077] ffff88014d79bfd8 0000000000015980 ffff88014d79bfd8 ffff8801d58316e0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190081] 0000000000015980 0000000000015980 ffff88014d79bfd8 0000000000015980 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190084] Call Trace: Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190095] [] io_schedule+0x73/0xc0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190099] [] get_request_wait+0xcc/0x1a0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190104] [] ? autoremove_wake_function+0x0/0x40 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190107] [] __make_request+0x93/0x4b0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190111] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190113] [] generic_make_request+0x1b3/0x540 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190116] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190121] [] ? bvec_alloc_bs+0x62/0x110 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190125] [] ? kmem_cache_alloc+0x77/0x120 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190127] [] submit_bio+0x82/0x110 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190131] [] ? __mark_inode_dirty+0x42/0x1d0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190173] [] xfs_submit_ioend_bio+0x57/0x90 [xfs] Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190189] [] xfs_submit_ioend+0xb2/0x110 [xfs] Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190205] [] xfs_page_state_convert+0x348/0x6d0 [xfs] Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190222] [] xfs_vm_writepage+0x95/0x180 [xfs] Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190225] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190230] [] __writepage+0x17/0x40 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190233] [] write_cache_pages+0x1c7/0x3d0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190235] [] ? __writepage+0x0/0x40 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190239] [] generic_writepages+0x24/0x30 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190254] [] xfs_vm_writepages+0x5d/0x80 [xfs] Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190257] [] do_writepages+0x21/0x40 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190260] [] writeback_single_inode+0xe6/0x3f0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190264] [] writeback_sb_inodes+0x195/0x280 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190266] [] writeback_inodes_wb+0xa0/0x1b0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190269] [] wb_writeback+0x24b/0x2b0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190274] [] ? del_timer_sync+0x22/0x30 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190277] [] wb_do_writeback+0xa9/0x190 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190279] [] ? process_timeout+0x0/0x10 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190282] [] bdi_writeback_task+0x53/0x160 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190285] [] ? bit_waitqueue+0x17/0xd0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190289] [] bdi_start_fn+0x86/0x100 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190292] [] ? bdi_start_fn+0x0/0x100 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190294] [] kthread+0x96/0xa0 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190298] [] kernel_thread_helper+0x4/0x10 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190301] [] ? retint_restore_args+0x5/0x6 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190304] [] ? kernel_thread_helper+0x0/0x10 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190307] INFO: task xfsbufd/md0:1323 blocked for more than 120 seconds. Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190313] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190319] xfsbufd/md0 D ffff880003e7d980 0 1323 2 0x00000000 Dec 5 23:04:48 ip-10-234-243-114 kernel: [ 3840.190323] ffff8801d761fb50 0000000000000246 ffffffff00000000 0000000000015980 ...skipping... Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200074] ffff8801d483bfd8 0000000000015980 ffff8801d483bfd8 ffff8801d5b10000 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200079] 0000000000015980 0000000000015980 ffff8801d483bfd8 0000000000015980 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200084] Call Trace: Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200096] [] ? _raw_spin_unlock_irqrestore+0x1e/0x30 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200134] [] xlog_state_get_iclog_space+0xe7/0x2d0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200140] [] ? default_wake_function+0x0/0x20 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200167] [] xlog_write+0x174/0x510 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200172] [] ? generic_make_request+0x1b3/0x540 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200176] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200191] [] xfs_log_write+0x77/0xa0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200209] [] xfs_trans_commit_iclog+0x169/0x300 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200225] [] ? xlog_grant_log_space+0x3f4/0x5d0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200241] [] ? xfs_log_reserve+0x106/0x170 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200256] [] _xfs_trans_commit+0x8d/0x2c0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200273] [] xfs_commit_dummy_trans+0x9e/0xf0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200289] [] xfs_sync_worker+0x74/0x80 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200305] [] xfssyncd+0x183/0x230 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200320] [] ? xfssyncd+0x0/0x230 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200324] [] kthread+0x96/0xa0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200327] [] kernel_thread_helper+0x4/0x10 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200331] [] ? retint_restore_args+0x5/0x6 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200334] [] ? kernel_thread_helper+0x0/0x10 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200351] INFO: task flush-9:0:1078 blocked for more than 120 seconds. Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200357] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200364] flush-9:0 D ffff880003e5f980 0 1078 2 0x00000000 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200367] ffff8801737ad640 0000000000000246 0000000000000000 0000000000015980 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200371] ffff8801737adfd8 0000000000015980 ffff8801737adfd8 ffff8801d75f96e0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200375] 0000000000015980 0000000000015980 ffff8801737adfd8 0000000000015980 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200378] Call Trace: Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200381] [] io_schedule+0x73/0xc0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200384] [] get_request_wait+0xcc/0x1a0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200387] [] ? autoremove_wake_function+0x0/0x40 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200389] [] __make_request+0x93/0x4b0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200392] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200395] [] generic_make_request+0x1b3/0x540 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200398] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200402] [] ? bvec_alloc_bs+0x62/0x110 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200406] [] ? kmem_cache_alloc+0x77/0x120 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200408] [] submit_bio+0x82/0x110 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200412] [] ? __mark_inode_dirty+0x42/0x1d0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200427] [] xfs_submit_ioend_bio+0x57/0x90 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200441] [] xfs_submit_ioend+0xb2/0x110 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200456] [] xfs_page_state_convert+0x348/0x6d0 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200471] [] xfs_vm_writepage+0x95/0x180 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200474] [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200478] [] __writepage+0x17/0x40 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200481] [] write_cache_pages+0x1c7/0x3d0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200483] [] ? __writepage+0x0/0x40 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200487] [] generic_writepages+0x24/0x30 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200501] [] xfs_vm_writepages+0x5d/0x80 [xfs] Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200504] [] do_writepages+0x21/0x40 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200507] [] writeback_single_inode+0xe6/0x3f0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200510] [] writeback_sb_inodes+0x195/0x280 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200513] [] writeback_inodes_wb+0xa0/0x1b0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200516] [] wb_writeback+0x24b/0x2b0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200520] [] ? del_timer_sync+0x22/0x30 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200523] [] wb_do_writeback+0xa9/0x190 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200526] [] ? process_timeout+0x0/0x10 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200529] [] bdi_writeback_task+0x53/0x160 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200531] [] ? bit_waitqueue+0x17/0xd0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200535] [] bdi_start_fn+0x86/0x100 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200538] [] ? bdi_start_fn+0x0/0x100 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200540] [] kthread+0x96/0xa0 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200542] [] kernel_thread_helper+0x4/0x10 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200546] [] ? retint_restore_args+0x5/0x6 Dec 6 00:36:00 ip-10-235-98-89 kernel: [ 3960.200548] [] ? kernel_thread_helper+0x0/0x10