Comment 8 for bug 606981

Ilia,

Thanks. I just have not understood this is the outcome of the discussion
on the XFS developers list. We will see to change tar4ibd to use same
IO mode as the MySQL running on files which is probably right solution

What is about in consistence - what kind of inconsistency one would observe
in this case ? Xtrabackup in general should be rather tolerant to it
(re-reading on bad
checksums)

On Sun, Aug 1, 2010 at 11:00 PM, Ilia Mirkin <email address hidden>wrote:

> I'm pretty sure that all the XFS devs are in agreement that fixing the
> hang is something that needs to be done. I brought up removing the
> warning earlier on LKML, and at least Dave Chinner's position was that
> it was best left in. [Not sure if Andrew Platts's warning is the same as
> mine, as it comes from a considerably older kernel.]
>
> However the claim that the XFS devs make is that mixing mysqld's
> O_DIRECT and the mechanism that tar4ibd uses to read the files in will
> result in inconsistent data being read in for any filesystem (and also
> exacerbates the XFS problem).
>
> --
> linux + o_direct + stream backup = broken?
> https://bugs.launchpad.net/bugs/606981
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona-XtraBackup.
>
> Status in Open source backup tool for InnoDB and XtraDB: New
>
> Bug description:
> I'm running mysql-5.0.90, xtrabackup 1.2, and have the following two
> options set relating to innodb flushing:
>
> innodb_flush_log_at_trx_commit=2
> innodb_flush_method=O_DIRECT
>
> Every time we run a (streaming) backup, we see many instances of
>
> [170371.562926] ------------[ cut here ]------------
> [170371.562930] WARNING: at fs/xfs/linux-2.6/xfs_lrw.c:651
> xfs_write+0x3dc/0x784()
> [170371.562931] Hardware name: PowerEdge R710
> [170371.562932] Modules linked in: cifs iTCO_wdt iTCO_vendor_support
> [170371.562936] Pid: 4322, comm: mysqld Tainted: G W
> 2.6.33-gentoo-r2 #1
> [170371.562937] Call Trace:
> [170371.562940] [<ffffffff8120da46>] ? xfs_write+0x3dc/0x784
> [170371.562943] [<ffffffff81038733>] warn_slowpath_common+0x77/0xa4
> [170371.562945] [<ffffffff8103876f>] warn_slowpath_null+0xf/0x11
> [170371.562947] [<ffffffff8120da46>] xfs_write+0x3dc/0x784
> [170371.562950] [<ffffffff81034825>] ? wake_up_state+0xb/0xd
> [170371.562953] [<ffffffff8120a51a>] xfs_file_aio_write+0x5a/0x5c
> [170371.562955] [<ffffffff810d7cbe>] do_sync_write+0xc0/0x106
> [170371.562959] [<ffffffff810d8624>] vfs_write+0xab/0x105
> [170371.562962] [<ffffffff810d86da>] sys_pwrite64+0x5c/0x7d
> [170371.562966] [<ffffffff81002a6b>] system_call_fastpath+0x16/0x1b
> [170371.562969] ---[ end trace 915525cd726ee99f ]---
>
> in the logs.
>
> The XFS guys say that this is a warning about bad O_DIRECT + not O_DIRECT
> interactions:
>
> http://lkml.org/lkml/2010/5/23/50 [The thread was started by someone else
> seeing the same warning, but I joined in later]
>
> Also, we had a filesystem hang, which was suggested to have been caused by
> this same problem:
>
> http://oss.sgi.com/archives/xfs/2010-07/msg00163.html
>
> Basically they claim that xtrabackup/tar4ibd is doing something that can
> break on any linux filesystem. Perhaps tar4ibd isn't using O_DIRECT when it
> should be? [Or it could be xtrabackup, but the warnings never reference the
> xtrabackup program directly, and it wasn't in the list of stuck tasks in the
> hang, hence my conclusion that it's tar4ibd]
>
>
>

--
Peter Zaitsev, CEO, Percona Inc.
Tel: +1 888 401 3401 ext 501 Skype: peter_zaitsev
24/7 Emergency Line +1 888 401 3401 ext 911

Percona Training Workshops
http://www.percona.com/training/