Jeremy,

Yes I think this is what we should look into doing.  I also do not think it
will be hard work.

We want to work around this issue on our side - absolutely but It is also
good if this problem is fixed on XFS
side... as it is general issue which can be triggered by other apps.

On Fri, Jul 30, 2010 at 6:58 PM, Jeremy Cole <email address hidden>wrote:

> We've also been seeing this exact same "BUG" message logged.  After some
> hair pulling we were able to track it down to xtrabackup with
> --stream=tar mode. I think the tar-streaming mode is fundamentally
> broken due to this. Maybe it was the original cause of the corruption
> complaints which triggered the creation of tar4ibd as well? Mixing of
> direct (InnoDB) and buffered (tar) IO is always dangerous.
>
> Peter: How hard would it be to add support to tar4ibd to use O_DIRECT if
> InnoDB is configured for it (which you already know from the rest of the
> configuration). It seems like this would be the best solution, and I
> don't think it would require much work.
>
> --
> linux + o_direct + stream backup = broken?
> https://bugs.launchpad.net/bugs/606981
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona-XtraBackup.
>
> Status in Open source backup tool for InnoDB and XtraDB: New
>
> Bug description:
> I'm running mysql-5.0.90, xtrabackup 1.2, and have the following two
> options set relating to innodb flushing:
>
> innodb_flush_log_at_trx_commit=2
> innodb_flush_method=O_DIRECT
>
> Every time we run a (streaming) backup, we see many instances of
>
> [170371.562926] ------------[ cut here ]------------
> [170371.562930] WARNING: at fs/xfs/linux-2.6/xfs_lrw.c:651
> xfs_write+0x3dc/0x784()
> [170371.562931] Hardware name: PowerEdge R710
> [170371.562932] Modules linked in: cifs iTCO_wdt iTCO_vendor_support
> [170371.562936] Pid: 4322, comm: mysqld Tainted: G        W
>  2.6.33-gentoo-r2 #1
> [170371.562937] Call Trace:
> [170371.562940]  [<ffffffff8120da46>] ? xfs_write+0x3dc/0x784
> [170371.562943]  [<ffffffff81038733>] warn_slowpath_common+0x77/0xa4
> [170371.562945]  [<ffffffff8103876f>] warn_slowpath_null+0xf/0x11
> [170371.562947]  [<ffffffff8120da46>] xfs_write+0x3dc/0x784
> [170371.562950]  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
> [170371.562953]  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
> [170371.562955]  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
> [170371.562959]  [<ffffffff810d8624>] vfs_write+0xab/0x105
> [170371.562962]  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
> [170371.562966]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
> [170371.562969] ---[ end trace 915525cd726ee99f ]---
>
> in the logs.
>
> The XFS guys say that this is a warning about bad O_DIRECT + not O_DIRECT
> interactions:
>
> http://lkml.org/lkml/2010/5/23/50 [The thread was started by someone else
> seeing the same warning, but I joined in later]
>
> Also, we had a filesystem hang, which was suggested to have been caused by
> this same problem:
>
> http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
>
> Basically they claim that xtrabackup/tar4ibd is doing something that can
> break on any linux filesystem. Perhaps tar4ibd isn't using O_DIRECT when it
> should be? [Or it could be xtrabackup, but the warnings never reference the
> xtrabackup program directly, and it wasn't in the list of stuck tasks in the
> hang, hence my conclusion that it's tar4ibd]
>
>
>


-- 
Peter Zaitsev, CEO, Percona Inc.
Tel: +1 888 401 3401 ext 501   Skype:  peter_zaitsev
24/7 Emergency Line +1 888 401 3401 ext 911

Percona Training Workshops
http://www.percona.com/training/