Combining direct_io and non-direct_io leads to hang

[Avleen Vig]

Version 0.12.2 - I know this isn't the latest but I looked through the changelogs and couldn't find any references to this being fixed.

This is related to issues that many apps have, and is described in more detail here:
    http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
and:
    https://bugs.launchpad.net/percona-xtrabackup/+bug/606981

When using both direct io and buffered io on the same file, you can cause corruption on the filesystem and other issues.
XFS out right hangs, but the problem exists on ext3 and other filesystems which silently carry on.

**this is a data corruption issue**.

This is the full stack trace we got from a recent hang:

Feb 22 19:55:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 19:55:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 19:55:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 19:55:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 19:55:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 19:55:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 19:55:14 virt11 kernel: Call Trace:
Feb 22 19:55:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 19:55:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 19:55:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 19:55:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 19:55:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 19:55:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 19:55:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 19:55:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 19:55:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 19:55:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 19:55:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 19:55:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 19:55:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 19:55:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 19:55:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 19:55:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 19:55:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 19:55:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 19:55:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 19:55:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 19:57:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 19:57:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 19:57:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 19:57:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 19:57:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 19:57:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 19:57:14 virt11 kernel: Call Trace:
Feb 22 19:57:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 19:57:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 19:57:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 19:57:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 19:57:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 19:57:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 19:57:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 19:57:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 19:57:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 19:57:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 19:57:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 19:57:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 19:57:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 19:57:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 19:57:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 19:57:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 19:57:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 19:57:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 19:57:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 19:57:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 19:59:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 19:59:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 19:59:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 19:59:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 19:59:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 19:59:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 19:59:14 virt11 kernel: Call Trace:
Feb 22 19:59:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 19:59:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 19:59:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 19:59:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 19:59:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 19:59:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 19:59:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 19:59:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 19:59:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 19:59:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 19:59:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 19:59:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 19:59:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 19:59:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 19:59:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 19:59:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 19:59:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 19:59:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 19:59:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 19:59:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:01:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:01:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:01:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:01:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:01:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:01:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:01:14 virt11 kernel: Call Trace:
Feb 22 20:01:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:01:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:01:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:01:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:01:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:01:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:01:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:01:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:01:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:01:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:01:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:01:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:01:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:01:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:01:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:01:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:01:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:01:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:01:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:01:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:03:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:03:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:03:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:03:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:03:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:03:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:03:14 virt11 kernel: Call Trace:
Feb 22 20:03:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:03:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:03:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:03:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:03:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:03:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:03:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:03:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:03:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:03:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:03:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:03:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:03:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:03:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:03:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:03:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:03:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:03:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:03:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:03:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:05:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:05:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:05:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:05:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:05:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:05:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:05:14 virt11 kernel: Call Trace:
Feb 22 20:05:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:05:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:05:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:05:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:05:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:05:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:05:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:05:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:05:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:05:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:05:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:05:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:05:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:05:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:05:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:05:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:05:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:05:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:05:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:05:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:07:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:07:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:07:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:07:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:07:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:07:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:07:14 virt11 kernel: Call Trace:
Feb 22 20:07:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:07:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:07:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:07:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:07:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:07:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:07:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:07:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:07:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:07:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:07:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:07:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:07:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:07:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:07:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:07:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:07:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:07:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:07:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:07:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:09:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:09:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:09:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:09:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:09:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:09:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:09:14 virt11 kernel: Call Trace:
Feb 22 20:09:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:09:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:09:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:09:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:09:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:09:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:09:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:09:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:09:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:09:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:09:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:09:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:09:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:09:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:09:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:09:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:09:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:09:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:09:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:09:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:11:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:11:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:11:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:11:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:11:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:11:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:11:14 virt11 kernel: Call Trace:
Feb 22 20:11:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:11:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:11:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:11:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:11:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:11:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:11:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:11:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:11:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:11:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:11:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:11:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:11:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:11:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:11:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:11:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:11:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:11:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:11:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:11:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Feb 22 20:13:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more than 120 seconds.
Feb 22 20:13:14 virt11 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 22 20:13:14 virt11 kernel: qemu-kvm D 0000000000000002 0 18360 1 0x00000080
Feb 22 20:13:14 virt11 kernel: ffff8808614457e8 0000000000000086 ffff8800282d5f80 ffff8800282d5f80
Feb 22 20:13:14 virt11 kernel: ffff880861445768 ffffffff810573ce ffff880861445768 0000000000000086
Feb 22 20:13:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8 000000000000f598 ffff8808f15bba78
Feb 22 20:13:14 virt11 kernel: Call Trace:
Feb 22 20:13:14 virt11 kernel: [<ffffffff810573ce>] ? activate_task+0x2e/0x40
Feb 22 20:13:14 virt11 kernel: [<ffffffff814dd755>] rwsem_down_failed_common+0x95/0x1d0
Feb 22 20:13:14 virt11 kernel: [<ffffffff814dd8b3>] rwsem_down_write_failed+0x23/0x30
Feb 22 20:13:14 virt11 kernel: [<ffffffff8126e573>] call_rwsem_down_write_failed+0x13/0x20
Feb 22 20:13:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
Feb 22 20:13:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa033ab02>] xfs_iomap+0x2e2/0x440 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa0354116>] __xfs_get_blocks+0x86/0x200 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa03542aa>] xfs_get_blocks_direct+0x1a/0x20 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffff811ac132>] __blockdev_direct_IO+0x872/0xc40
Feb 22 20:13:14 virt11 kernel: [<ffffffffa0353f50>] xfs_vm_direct_IO+0xb0/0xf0 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa0354290>] ? xfs_get_blocks_direct+0x0/0x20 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa0353cc0>] ? xfs_end_io_direct+0x0/0xe0 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffff8106dd57>] ? current_fs_time+0x27/0x30
Feb 22 20:13:14 virt11 kernel: [<ffffffff8110df22>] generic_file_direct_write+0xc2/0x190
Feb 22 20:13:14 virt11 kernel: [<ffffffffa034bfcf>] ? xfs_trans_unlocked_item+0x4f/0x60 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffffa035dd1d>] xfs_write+0x4fd/0xb70 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffff8105dc72>] ? default_wake_function+0x12/0x20
Feb 22 20:13:14 virt11 kernel: [<ffffffffa03599a1>] xfs_file_aio_write+0x61/0x70 [xfs]
Feb 22 20:13:14 virt11 kernel: [<ffffffff8117241a>] do_sync_write+0xfa/0x140
Feb 22 20:13:14 virt11 kernel: [<ffffffff8107fbc2>] ? send_signal+0x42/0x80
Feb 22 20:13:14 virt11 kernel: [<ffffffff8108e160>] ? autoremove_wake_function+0x0/0x40
Feb 22 20:13:14 virt11 kernel: [<ffffffff8107ff96>] ? group_send_sig_info+0x56/0x70
Feb 22 20:13:14 virt11 kernel: [<ffffffff81211d3b>] ? selinux_file_permission+0xfb/0x150
Feb 22 20:13:14 virt11 kernel: [<ffffffff812051a6>] ? security_file_permission+0x16/0x20
Feb 22 20:13:14 virt11 kernel: [<ffffffff81172718>] vfs_write+0xb8/0x1a0
Feb 22 20:13:14 virt11 kernel: [<ffffffff810d1b62>] ? audit_syscall_entry+0x272/0x2a0
Feb 22 20:13:14 virt11 kernel: [<ffffffff81173212>] sys_pwrite64+0x82/0xa0
Feb 22 20:13:14 virt11 kernel: [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b

[Stefan Hajnoczi (stefanha)]

On Wed, Feb 22, 2012 at 09:49:59PM -0000, Avleen Vig wrote:
> Version 0.12.2 - I know this isn't the latest but I looked through the
> changelogs and couldn't find any references to this being fixed.
>
> This is related to issues that many apps have, and is described in more detail here:
> http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
> and:
> https://bugs.launchpad.net/percona-xtrabackup/+bug/606981
>
> When using both direct io and buffered io on the same file, you can cause corruption on the filesystem and other issues.
> XFS out right hangs, but the problem exists on ext3 and other filesystems which silently carry on.
>
> **this is a data corruption issue**.
>
> This is the full stack trace we got from a recent hang:

I only see qemu-kvm in the backtrace. QEMU normally only access the
image file either with or without O_DIRECT - not both at the same time.
Were you running another program that accesses the image file at the
same time?

Stefan

[Avleen Vig (avleen)]

Nothing else at all accesses those files, which is one of the things that concerned me the most :-/
It looks like it call came from the same place.

[Stefan Hajnoczi (stefanha)]

On Thu, Feb 23, 2012 at 9:39 PM, Avleen Vig <email address hidden> wrote:
> Nothing else at all accesses those files, which is one of the things that concerned me the most :-/
> It looks like it call came from the same place.

It's possible to check the file descriptor flags using
/proc/$PID/fdinfo/$FD. So for example, you could find the qemu-kvm
process and list its open file descriptors using ls -l /proc/$(pgrep
qemu-kvm)/fd. Find the file descriptor to the image file and look up
its flags in /proc/$PID/fdinfo/$FD.

If the file descriptor has bit 040000 set then it is O_DIRECT. If
not, it uses buffered I/O.

Using this technique you could determine whether or not your system is
accessing the file using both direct and buffered I/O. It would also
tell you which processes are doing this.

Stefan

[Avleen Vig (avleen)]

Excellent. I will do this the next time I see this happen. It probably
won't be long.
On Feb 24, 2012 12:45 AM, "Stefan Hajnoczi" <email address hidden>
wrote:

> On Thu, Feb 23, 2012 at 9:39 PM, Avleen Vig <email address hidden>
> wrote:
> > Nothing else at all accesses those files, which is one of the things
> that concerned me the most :-/
> > It looks like it call came from the same place.
>
> It's possible to check the file descriptor flags using
> /proc/$PID/fdinfo/$FD. So for example, you could find the qemu-kvm
> process and list its open file descriptors using ls -l /proc/$(pgrep
> qemu-kvm)/fd. Find the file descriptor to the image file and look up
> its flags in /proc/$PID/fdinfo/$FD.
>
> If the file descriptor has bit 040000 set then it is O_DIRECT. If
> not, it uses buffered I/O.
>
> Using this technique you could determine whether or not your system is
> accessing the file using both direct and buffered I/O. It would also
> tell you which processes are doing this.
>
> Stefan
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/939027
>
> Title:
> Combining direct_io and non-direct_io leads to hang
>
> Status in QEMU:
> New
>
> Bug description:
> Version 0.12.2 - I know this isn't the latest but I looked through the
> changelogs and couldn't find any references to this being fixed.
>
> This is related to issues that many apps have, and is described in more
> detail here:
> http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
> and:
> https://bugs.launchpad.net/percona-xtrabackup/+bug/606981
>
> When using both direct io and buffered io on the same file, you can cause
> corruption on the filesystem and other issues.
> XFS out right hangs, but the problem exists on ext3 and other filesystems
> which silently carry on.
>
> **this is a data corruption issue**.
>
> This is the full stack trace we got from a recent hang:
>
> Feb 22 19:55:14 virt11 kernel: INFO: task qemu-kvm:18360 blocked for more
> than 120 seconds.
> Feb 22 19:55:14 virt11 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 22 19:55:14 virt11 kernel: qemu-kvm D 0000000000000002 0
> 18360 1 0x00000080
> Feb 22 19:55:14 virt11 kernel: ffff8808614457e8 0000000000000086
> ffff8800282d5f80 ffff8800282d5f80
> Feb 22 19:55:14 virt11 kernel: ffff880861445768 ffffffff810573ce
> ffff880861445768 0000000000000086
> Feb 22 19:55:14 virt11 kernel: ffff8808f15bba78 ffff880861445fd8
> 000000000000f598 ffff8808f15bba78
> Feb 22 19:55:14 virt11 kernel: Call Trace:
> Feb 22 19:55:14 virt11 kernel: [<ffffffff810573ce>] ?
> activate_task+0x2e/0x40
> Feb 22 19:55:14 virt11 kernel: [<ffffffff814dd755>]
> rwsem_down_failed_common+0x95/0x1d0
> Feb 22 19:55:14 virt11 kernel: [<ffffffff814dd8b3>]
> rwsem_down_write_failed+0x23/0x30
> Feb 22 19:55:14 virt11 kernel: [<ffffffff8126e573>]
> call_rwsem_down_write_failed+0x13/0x20
> Feb 22 19:55:14 virt11 kernel: [<ffffffff814dcdb2>] ? down_write+0x32/0x40
> Feb 22 19:55:14 virt11 kernel: [<ffffffffa0332d6e>] xfs_ilock+0x7e/0xd0
> [xfs]
> Feb 22 19:55:14 virt11 kernel: [<ffffff...

[Thomas Huth (th-huth)]

Triaging old bug tickets... can you still reproduce this issue with the latest version of QEMU? Or could we close this ticket nowadays?

[Avleen Vig (avleen)]

Unfortunately I don't have access to a qemu instance to test this any more :)