Andrew,

Thanks. This is helpful information.

As I understand the XFS guys would rather get some test case which may be
helpful
We also should see what can be done with Xtrabackup so it uses DIRECT_IO for
copy
so things are consistent.

On Thu, Jul 29, 2010 at 5:52 PM, Andrew Platts <email address hidden>wrote:

> Well,
>
> what we see is the slave lag  exceeds  our threshold during backups
> (causing MMM to take action) . So   I believe we're  getting a
> filesystem hang as the  first bug reporter mentioned.  This  happens
> just prior  to the  call trace. Also very simple  PK queries start
> taking  over 30 secs.
>
> This is debian 5.0.4 with xfs (Linux  2.6.26-2-amd64 #1 SMP Wed May 12
> 18:03:14 UTC 2010 x86_64 GNU/Linux)
>
> I see from   the link below that   this is not something the   file
> system guys think is their problem
>
> http://oss.sgi.com/archives/xfs/2010-07/msg00184.html
>
>
> Dave Chinner says :
> "Right now I'm not particularly inclined to dig into this further;
> it's obvious the applications are doing something that is not
> supported (by XFS or the generic page cache code), so this is the
> first thing you really need to care about getting fixed if you value
> your backups..."
>
> The alternative right now for us might be to turn O_DIRECT off  - we're
> using SSDs  , not  sure if O_DIRECT will have a hugh impact in this case
> ...
>
> --
> linux + o_direct + stream backup = broken?
> https://bugs.launchpad.net/bugs/606981
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona-XtraBackup.
>
> Status in Open source backup tool for InnoDB and XtraDB: New
>
> Bug description:
> I'm running mysql-5.0.90, xtrabackup 1.2, and have the following two
> options set relating to innodb flushing:
>
> innodb_flush_log_at_trx_commit=2
> innodb_flush_method=O_DIRECT
>
> Every time we run a (streaming) backup, we see many instances of
>
> [170371.562926] ------------[ cut here ]------------
> [170371.562930] WARNING: at fs/xfs/linux-2.6/xfs_lrw.c:651
> xfs_write+0x3dc/0x784()
> [170371.562931] Hardware name: PowerEdge R710
> [170371.562932] Modules linked in: cifs iTCO_wdt iTCO_vendor_support
> [170371.562936] Pid: 4322, comm: mysqld Tainted: G        W
>  2.6.33-gentoo-r2 #1
> [170371.562937] Call Trace:
> [170371.562940]  [<ffffffff8120da46>] ? xfs_write+0x3dc/0x784
> [170371.562943]  [<ffffffff81038733>] warn_slowpath_common+0x77/0xa4
> [170371.562945]  [<ffffffff8103876f>] warn_slowpath_null+0xf/0x11
> [170371.562947]  [<ffffffff8120da46>] xfs_write+0x3dc/0x784
> [170371.562950]  [<ffffffff81034825>] ? wake_up_state+0xb/0xd
> [170371.562953]  [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
> [170371.562955]  [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
> [170371.562959]  [<ffffffff810d8624>] vfs_write+0xab/0x105
> [170371.562962]  [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
> [170371.562966]  [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
> [170371.562969] ---[ end trace 915525cd726ee99f ]---
>
> in the logs.
>
> The XFS guys say that this is a warning about bad O_DIRECT + not O_DIRECT
> interactions:
>
> http://lkml.org/lkml/2010/5/23/50 [The thread was started by someone else
> seeing the same warning, but I joined in later]
>
> Also, we had a filesystem hang, which was suggested to have been caused by
> this same problem:
>
> http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
>
> Basically they claim that xtrabackup/tar4ibd is doing something that can
> break on any linux filesystem. Perhaps tar4ibd isn't using O_DIRECT when it
> should be? [Or it could be xtrabackup, but the warnings never reference the
> xtrabackup program directly, and it wasn't in the list of stuck tasks in the
> hang, hence my conclusion that it's tar4ibd]
>
>
>


-- 
Peter Zaitsev, CEO, Percona Inc.
Tel: +1 888 401 3401 ext 501   Skype:  peter_zaitsev
24/7 Emergency Line +1 888 401 3401 ext 911

Percona Training Workshops
http://www.percona.com/training/