linux + o_direct + stream backup = broken?
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB |
Fix Released
|
High
|
Alexey Kopytov |
Bug Description
I'm running mysql-5.0.90, xtrabackup 1.2, and have the following two options set relating to innodb flushing:
innodb_
innodb_
Every time we run a (streaming) backup, we see many instances of
[170371.562926] ------------[ cut here ]------------
[170371.562930] WARNING: at fs/xfs/
[170371.562931] Hardware name: PowerEdge R710
[170371.562932] Modules linked in: cifs iTCO_wdt iTCO_vendor_support
[170371.562936] Pid: 4322, comm: mysqld Tainted: G W 2.6.33-gentoo-r2 #1
[170371.562937] Call Trace:
[170371.562940] [<ffffffff8120d
[170371.562943] [<ffffffff81038
[170371.562945] [<ffffffff81038
[170371.562947] [<ffffffff8120d
[170371.562950] [<ffffffff81034
[170371.562953] [<ffffffff8120a
[170371.562955] [<ffffffff810d7
[170371.562959] [<ffffffff810d8
[170371.562962] [<ffffffff810d8
[170371.562966] [<ffffffff81002
[170371.562969] ---[ end trace 915525cd726ee99f ]---
in the logs.
The XFS guys say that this is a warning about bad O_DIRECT + not O_DIRECT interactions:
http://
Also, we had a filesystem hang, which was suggested to have been caused by this same problem:
http://
Basically they claim that xtrabackup/tar4ibd is doing something that can break on any linux filesystem. Perhaps tar4ibd isn't using O_DIRECT when it should be? [Or it could be xtrabackup, but the warnings never reference the xtrabackup program directly, and it wasn't in the list of stuck tasks in the hang, hence my conclusion that it's tar4ibd]
Related branches
- Vadim Tkachenko: Approve
-
Diff: 12 lines (+1/-1)1 file modifiedxtrabackup.c (+1/-1)
Changed in percona-xtrabackup: | |
assignee: | nobody → Yasufumi Kinoshita (yasufumi-kinoshita) |
Changed in percona-xtrabackup: | |
status: | Confirmed → In Progress |
Changed in percona-xtrabackup: | |
status: | In Progress → Fix Committed |
Changed in percona-xtrabackup: | |
status: | Fix Committed → Fix Released |
We also experience this bug
Also using O_DIRECT with innodb_ flush_log_ at_trx_ commit= 2
With the following command xtrabackup. log| gzip - | nc -vw60 $target $port
innobackupex-1.5.1 --slave-info --stream=tar . 2> /tmp/$source.
Sample output from /var/log/messages
[6896419.396214] ------------[ cut here ]------------ linux-2. 6/xfs_lrw. c:726 xfs_write+ 0x3f5/0x722 [xfs]() 930>] warn_on_ slowpath+ 0x51/0x7a 997>] check_preempt_ wakeup+ 0xc4/0xf0 157>] try_to_ wake_up+ 0x118/0x129 230>] __up_write+ 0x82/0x10e c68>] :xfs:xfs_ iunlock+ 0x42/0x7c 6cc>] :xfs:xfs_ write+0x3f5/ 0x722 3e0>] enqueue_ hrtimer+ 0xd7/0xe4 e83>] do_sync_ write+0xc9/ 0x10c 171>] autoremove_ wake_function+ 0x0/0x2e 62d>] vfs_write+ 0xad/0x156 2ee>] fget_light+ 0x4f/0x82 cb8>] sys_pwrite64+ 0x50/0x70 267>] ptregscall_ common+ 0x67/0xb0 eda>] system_ call_after_ swapgs+ 0x8a/0x8f
[6896419.396214] WARNING: at fs/xfs/
[6896419.396214] Modules linked in: ipv6 xfs loop snd_pcm snd_timer snd soundcore snd_page_alloc i2c_i801 psmouse pcspkr i2c_core serio_raw button joydev evdev ext3 jbd mbcache dm_mirror dm_log dm_snapshot dm_mod raid1 raid0 md_mod sd_mod sg sr_mod cdrom usbhid hid ff_memless mptsas mptscsih mptbase ahci scsi_transport_sas libata scsi_mod dock ehci_hcd uhci_hcd igb thermal processor fan thermal_sys [last unloaded: scsi_wait_scan]
[6896419.396214] Pid: 6909, comm: mysqld Tainted: G W 2.6.26-2-amd64 #1
[6896419.396214]
[6896419.396214] Call Trace:
[6896419.396214] [<ffffffff80234
[6896419.396214] [<ffffffff80231
[6896419.396214] [<ffffffff8022c
[6896419.396214] [<ffffffff8031e
[6896419.396214] [<ffffffffa022e
[6896419.396214] [<ffffffffa0252
[6896419.396214] [<ffffffff80248
[6896419.396223] [<ffffffff8029a
[6896419.396231] [<ffffffff80246
[6896419.396236] [<ffffffff8029b
[6896419.396236] [<ffffffff8029c
[6896419.396236] [<ffffffff8029b
[6896419.396236] [<ffffffff8020c
[6896419.396236] [<ffffffff8020b
[6896419.396236]