XtraBackup 2.1.2 Hangs on SST

Bug #1182698 reported by Jervin R
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Critical
Alexey Kopytov
2.1
Fix Released
Critical
Alexey Kopytov
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Fix Released
Undecided
Unassigned

Bug Description

OS: Ubuntu 12.04 x64
XtraBackup 2.1.2 installed

root@upxdbc01:/var/lib/mysql# dpkg -l|grep percona
ii percona-xtrabackup 2.1.2-611.precise Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.30-23.7.4-405.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.30-23.7.4-405.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 150.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.30-23.7.4-405.precise Percona Server database server binaries

Related wsrep settings:

wsrep_node_name = uxdbc01
wsrep_cluster_name = uxdbc
wsrep_debug = 1
wsrep_node_address = 192.168.56.53
wsrep_node_incoming_address = 192.168.56.53
wsrep_provider = /usr/lib/libgalera_smm.so
wsrep_provider_options = "ist.recv_addr=192.168.56.53:4568;gmcast.listen_addr=tcp://192.168.56.53:4567;"
wsrep_slave_threads = 2
wsrep_sst_method = xtrabackup
wsrep_sst_receive_address = 192.168.56.53
wsrep_cluster_address = gcomm://

The second node is identical excep on some obvious wsrep_config values. AN strace of the innobackupex process shows the keepalive ping failing:

root 7489 0.0 0.0 4400 760 pts/0 S 20:57 0:00 /bin/sh /usr/bin/mysqld_safe
mysql 8292 0.2 9.9 1496028 101084 pts/0 Sl 20:57 0:00 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --open-files-limit=8192 --pid-file=/var/lib/m
mysql 8411 0.0 0.0 4400 612 pts/0 S 20:58 0:00 \_ sh -c wsrep_sst_xtrabackup --role 'donor' --address '192.168.56.54:4444/xtrabackup_sst' --auth '(null)' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-f
mysql 8412 0.0 0.1 20816 1732 pts/0 S 20:58 0:00 \_ /bin/bash -ue /usr//bin/wsrep_sst_xtrabackup --role donor --address 192.168.56.54:4444/xtrabackup_sst --auth (null) --socket /var/run/mysqld/mysqld.sock --datadir /var/lib/mysql/
mysql 8423 0.0 1.4 145684 14964 pts/0 S 20:58 0:00 \_ perl /usr//bin/innobackupex --galera-info --stream=tar --defaults-file=/etc/my.cnf --socket=/var/run/mysqld/mysqld.sock /tmp
mysql 8426 0.0 1.1 145684 12192 pts/0 S 20:58 0:00 | \_ perl /usr//bin/innobackupex --galera-info --stream=tar --defaults-file=/etc/my.cnf --socket=/var/run/mysqld/mysqld.sock /tmp
mysql 8424 0.0 0.0 16492 912 pts/0 S 20:58 0:00 \_ nc 192.168.56.54 4444
root@upxdbc01:/var/lib/mysql# strace -p 8426
Process 8426 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(4, "\34\0\0\0\3select 'xtrabackup ping 61'", 32) = 32
read(4, "\1\0\0\1\1(\0\0\2\3def\0\0\0\22xtrabackup ping"..., 16384) = 90
rt_sigprocmask(SIG_BLOCK, [CHLD], [HUP INT QUIT PIPE ALRM TERM TSTP], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE ALRM TERM TSTP], NULL, 8) = 0
nanosleep({5, 0}, 0x7fff681915a0) = 0
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(4, "\34\0\0\0\3select 'xtrabackup ping 62'", 32) = 32
read(4, "\1\0\0\1\1(\0\0\2\3def\0\0\0\22xtrabackup ping"..., 16384) = 90
rt_sigprocmask(SIG_BLOCK, [CHLD], [HUP INT QUIT PIPE ALRM TERM TSTP], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE ALRM TERM TSTP], NULL, 8) = 0
nanosleep({5, 0}, 0x7fff681915a0) = 0

Running the innobackupex command manually as root or mysql user seems to work fine.

Related branches

Revision history for this message
Jervin R (revin) wrote :

Another strace output from customer:

poll([{fd=22, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(22, "\34\0\0\0\3select 'xtrabackup ping 15'", 32) = 32
read(22, "\1\0\0\1\1(\0\0\2\3def\0\0\0\22xtrabackup ping"..., 16384) = 90
write(21, "main::start_keepalives(/usr/bin/"..., 85) = 85
write(21, "\33[4m[pid=17664->17669] DB<1> \33["..., 39) = 39
read(11, 0x7fdeb6cf7000, 8192) = -1 EIO (Input/output error)
write(21, "\33[0m", 4) = 4
write(21, "main::start_keepalives(/usr/bin/"..., 91) = 91
write(21, "\33[4m[pid=17664->17669] DB<1> \33["..., 39) = 39
read(11, 0x7fdeb6cf7000, 8192) = -1 EIO (Input/output error)

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Another related gist posted from a freenode: https://gist.github.com/anonymous/0e9fea54741e4b2ac093

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Able to confirm with latest PXC and PXB 2.1.2

Changed in percona-xtrabackup:
status: New → Confirmed
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Sprunge log of strace: http://sprunge.us/ZQDi

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

http://sprunge.us/eOfC -- innobackup log

Note: Xtrabackup isn't forked yet.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

For anyone to test, following diff (from XB dev) works: https://gist.github.com/ronin13/ace49de95cd87b68758d

Revision history for this message
Denis Alligand (d-alligand) wrote :

from which innobackupex do i need to apply the patch?

whith current version i do have the following error

patch -p0 < innobackupex.patch
patching file /usr/bin/innobackupex
Hunk #1 FAILED at 285.
Hunk #2 FAILED at 1404.
2 out of 2 hunks FAILED -- saving rejects to file /usr/bin/innobackupex.rej

or

patch -p1 < /root/innobackupex.patch
patching file tmp/innobackupex
Hunk #1 FAILED at 285.
Hunk #2 FAILED at 1404.
2 out of 2 hunks FAILED -- saving rejects to file tmp/innobackupex.rej

version I do use:

dpkg -l|grep percona
ii percona-toolkit 2.2.2 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.0.6-521.squeeze Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.30-23.7.4-405.squeeze Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.30-23.7.4-405.squeeze Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 150.squeeze Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.30-23.7.4-405.squeeze Percona Server database server binaries

/usr/bin/innobackupex --version
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Denis

Revision history for this message
Denis Alligand (d-alligand) wrote :

i managed to fetch the file from the bzr repo,

patched version seems to work, will make some more extended test on it.

Regards,

Denis

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Patch to be applied on innobackupex in PXB 2.1.2

I have also added a testcase for this - https://bugs.launchpad.net/percona-xtrabackup/+bug/1182995

Changed in percona-xtradb-cluster:
status: New → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1354

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-244

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.