Bug #1707918 “using innobackupx with --slave-info output file xt...” : Bugs : Percona XtraBackup moved to https://jira.percona.com/projects/PXB

Revision history for this message

Jericho Rivera (jericho-rivera) wrote on 2017-08-02:

#1

[root@centos2 ~]# cat xtrabackup_binlog_info
mysql-bin.000005 154 4879c9cd-5318-11e7-b3cd-00163e3fe841:1-42754
[root@centos2 ~]#
[root@centos2 ~]# xtrabackup --version
xtrabackup version 2.4.8 based on MySQL server 5.7.13 Linux (x86_64) (revision id: 97330f7)
[root@centos2 ~]#
[root@centos2 ~]# more xtrabackup_slave_info
SET
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[root@centos2 ~]# more xtrabackup_info
uuid = 99d082e1-774c-11e7-bb0b-00163e4f04f3
name =
tool_name = innobackupex
tool_command = --user=root --socket=/tmp/mysql.sock --slave-info --stream=tar ./
tool_version = 2.4.8
ibbackup_version = 2.4.8
server_version = 5.7.18-15-log
start_time = 2017-08-02 02:33:56
end_time = 2017-08-02 02:34:13
lock_time = 0
binlog_pos = filename 'mysql-bin.000005', position '154', GTID of the last change '4879c9cd-5318-11e7-b3cd-00163e3fe841:1-42754'
innodb_from_lsn = 0
innodb_to_lsn = 5061435112
partial = N
incremental = N
format = tar
compact = N
compressed = N
encrypted = N

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2017-08-02:

#2

Tested with 2.3.9 on PS 5.6 slave server:

xtrabackup --defaults-file=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_6_36/node1/my.sandbox.cnf --user=msandbox --password='msandbox' --target-dir=/home/shahriyar.rzaev/backup_dir/ps_5.6_master/full/2017-08-02_08-54-58 --backup --host=127.0.0.1 --port=23902 --binlog-info=AUTO --galera-info --parallel 4 --check-privileges --no-version-check --slave-info

$ cat xtrabackup_slave_info
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000019', MASTER_LOG_POS=2555

Tested with 2.4.8 on PS 5.6 slave server with same backup command:

$ cat xtrabackup_slave_info
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000019', MASTER_LOG_POS=2555;

Same result with innobackupex command:

/home/shahriyar.rzaev/Percona_Xtrabackups/target/percona-xtrabackup-2.4.x-debug/bin/innobackupex -umsandbox -pmsandbox --socket=/tmp/mysql_sandbox23902.sock --slave-info --datadir=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_6_36/node1/data /home/shahriyar.rzaev/backup_dir/ps_5.6_master/full/2017-08-02_08-57-44

$ cat xtrabackup_slave_info
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000019', MASTER_LOG_POS=2555;

Revision history for this message

Rikard Eriksson (rikard-eriksson) wrote on 2017-08-02:

#3

Ah, sorry we are running Server 5.7

Percona-Server-5.7.17-11-Linux.x86_64.ssl101

Revision history for this message

Rikard Eriksson (rikard-eriksson) wrote on 2017-08-02:

#4

And looking at my xtrabackup_info the row for binlog is empty

binlog_pos =

Revision history for this message

Rikard Eriksson (rikard-eriksson) wrote on 2017-08-02:

#5

But Im taking the snapshot from a slave so it's the slave-info that is missing

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-02:

#6

This might relate to an issue we are seeing:

$ tar -tvf file.tar xtrabackup_slave_info
tar: Archive base-256 value is out of time_t range
-rw-rw---- 0/0 0 3708721243212117248 xtrabackup_slave_info
tar: Exiting with failure status due to previous errors

with 2.4.7, this looks as follows:

$ tar -tvf file.tar xtrabackup_slave_info
-rw-rw---- 0/0 380 2017-08-02 13:59 xtrabackup_slave_info

... the issue seems to have been introduced by the pretty large libarchive update in https://github.com/percona/percona-xtrabackup/commit/a1cb0b7ad6345534c09d05284189e41c574669f5

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-02:

#7

Btw: as this breaks the tarfile unpacking completely, all code expecting a positive tar unpack files.

Thus, I recommend giving this bug highest priority.

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-02:

#8

s/files/fails/ ...

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2017-08-02:

#9

@Rikard

Original bug report description:

"using innobackupx with --slave-info output file xtrabackup_slave_info does not contain slave information in 2.4.8[]"

Tested with 2.4.8 on PS 5.7.18 slave:

/home/shahriyar.rzaev/Percona_Xtrabackups/target/percona-xtrabackup-2.4.x-debug/bin/innobackupex -umsandbox -pmsandbox --socket=/tmp/mysql_sandbox20394.sock --slave-info --datadir=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_7_18/node1/data --stream=tar ./ > /home/shahriyar.rzaev/backup_dir/ps_5.7_master/full/2017-08-02_14-24-49/file.tar

$ cat xtrabackup_binlog_info
mysql-bin.000013 234 00020393-1111-1111-1111-111111111111:1-1511,
f9eac343-71f9-11e7-b64b-002590e9b448:1-3

$ cat xtrabackup_info
uuid = 312fe3ae-778f-11e7-8ed6-002590e9b448
name =
tool_name = innobackupex
tool_command = -umsandbox -pmsandbox --socket=/tmp/mysql_sandbox20394.sock --slave-info --datadir=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_7_18/node1/data --stream=tar ./
tool_version = 2.4.7
ibbackup_version = 2.4.7
server_version = 5.7.18-15-log
start_time = 2017-08-02 14:30:02
end_time = 2017-08-02 14:30:54
lock_time = 0
binlog_pos = filename 'mysql-bin.000013', position '234', GTID of the last change '00020393-1111-1111-1111-111111111111:1-1511,
f9eac343-71f9-11e7-b64b-002590e9b448:1-3'
innodb_from_lsn = 0
innodb_to_lsn = 992198332
partial = N
incremental = N
format = tar
compact = N
compressed = N
encrypted = N

$ cat xtrabackup_slave_info
SET GLOBAL gtid_purged='00020393-1111-1111-1111-111111111111:1-1511, f9eac343-71f9-11e7-b64b-002590e9b448:1-3';
CHANGE MASTER TO MASTER_AUTO_POSITION=1;

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2017-08-02:

#10

Here is the result what I have figured out with xtrabackup_slave_info file after using:

/home/shahriyar.rzaev/Percona_Xtrabackups/target/percona-xtrabackup-2.4.x-debug/bin/innobackupex -umsandbox -pmsandbox --socket=/tmp/mysql_sandbox20394.sock --slave-info --datadir=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_7_18/node1/data --stream=tar ./ > /home/shahriyar.rzaev/backup_dir/ps_5.7_master/full/2017-08-02_14-24-49/file.tar

$ cat xtrabackup_slave_info
SET GLOBAL gtid_purged='00020393-1111-1111-1111-111111111111:1-1511, f9eac343-71f9-11e7-b64b-002590e9b448:1-3';
CHANGE MASTER TO MASTER_AUTO_POSITION=1;
^C

$ strace cat xtrabackup_slave_info

read(3, "SET GLOBAL gtid_purged='00020393"..., 65536) = 65536
write(1, "SET GLOBAL gtid_purged='00020393"..., 65536SET GLOBAL gtid_purged='00020393-1111-1111-1111-111111111111:1-1511, f9eac343-71f9-11e7-b64b-002590e9b448:1-3';
CHANGE MASTER TO MASTER_AUTO_POSITION=1;
) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536

52M size + 1970 create date for this file:

$ ls -lthr
total 1,1G
-rw-rw---- 1 shahriyar.rzaev shahriyar.rzaev 52M Avq 2 1970 xtrabackup_slave_info

$ tar -tvf file.tar xtrabackup_slave_info
-rw-rw---- 0/0 53516497 1970-08-02 01:58 xtrabackup_slave_info

Here is the result what I have figured out with xtrabackup_slave_info file after using:

/home/shahriyar.rzaev/Percona_Xtrabackups/target/percona-xtrabackup-2.4.x-debug/bin/innobackupex -umsandbox -pmsandbox --socket=/tmp/mysql_sandbox20394.sock --slave-info --datadir=/home/shahriyar.rzaev/sandboxes/rsandbox_Percona-Server-5_7_18/node1/data --stream=tar ./ > /home/shahriyar.rzaev/backup_dir/ps_5.7_master/full/2017-08-02_14-24-49/file.tar

$ cat xtrabackup_slave_info
SET GLOBAL gtid_purged='00020393-1111-1111-1111-111111111111:1-1511, f9eac343-71f9-11e7-b64b-002590e9b448:1-3';
CHANGE MASTER TO MASTER_AUTO_POSITION=1;
^C

$ strace cat xtrabackup_slave_info

read(3, "SET GLOBAL gtid_purged='00020393"..., 65536) = 65536
write(1, "SET GLOBAL gtid_purged='00020393"..., 65536SET GLOBAL gtid_purged='00020393-1111-1111-1111-111111111111:1-1511, f9eac343-71f9-11e7-b64b-002590e9b448:1-3';
CHANGE MASTER TO MASTER_AUTO_POSITION=1;
) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536
write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65536) = 65536

52M size + 1970 create date for this file:

$ ls -lthr
total 1,1G
-rw-rw---- 1 shahriyar.rzaev shahriyar.rzaev   52M Avq  2  1970 xtrabackup_slave_info

$ tar -tvf file.tar xtrabackup_slave_info
-rw-rw---- 0/0        53516497 1970-08-02 01:58 xtrabackup_slave_info

Revision history for this message

Rikard Eriksson (rikard-eriksson) wrote on 2017-08-02:

#11

Might be my mistake that I'm missing some input to this but

Percona server: Percona-Server-5.7.17-11-Linux.x86_64.ssl101
xtrabackup: percona-xtrabackup-2.4.8-Linux-x86_64
CentOS: 7

I'm NOT using GTID, just the old default master->slave replication

I run the command on one of my slave, (NOT the master).

I run this command to create the backup:
/usr/local/percona-xtrabackup-2.4.8-Linux-x86_64/bin/innobackupex
--defaults-file=/db/mysql_main_slave/mysql_main_slave.cnf
--socket=/tmp/mysql_main_pl_s_d.sock
--user=root
--password=ThePassword
--slave-info --stream=tar ./ > file.tar

The backup ends with "completed OK!" and I have no problem to extract the tar archive.

What exists in the xtrabackup_slave_info file after the backup is:
CHAN

What I expect to be in the file, (xtrabackup_slave_info) is:
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000098', MASTER_LOG_POS=333969206

If I change to older version of the backup scripts "percona-xtrabackup-2.4.5-Linux-x86_64" then it works, the file contains the correct slave position information.

Revision history for this message

Sergei Glushchenko (sergei.glushchenko) wrote on 2017-08-02:

#12

Apparently, only streaming in TAR format is affected. As workaround you can use streaming in xbstream format. It has some benefits over TAR streaming as well. One of them is much higher speed.

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-02:

#13

tar combined with Zstandard compression (we're using zstd 1.2.0 with -T4 for multithreaded operation) is so far giving us the best results, and we'd love to stick with it, so - right now we're stuck at percona-xtrabackup 2.4.7 ... :)

Revision history for this message

Sergei Glushchenko (sergei.glushchenko) wrote on 2017-08-02:

#14

hmm... so, you compressing TAR produced by xtrabackup. You can compress xbstream the same way, right?

Revision history for this message

Sergei Glushchenko (sergei.glushchenko) wrote on 2017-08-02:

#15

For performance and benchmarks please see

https://www.percona.com/blog/2017/03/29/performance-evaluation-of-sst-data-transfer-without-encryption-part-1/
and
https://www.percona.com/blog/2017/03/30/performance-evaluation-of-sst-data-transfer-with-encryption-part-2/

Note, that xtrabackup streaming in tar format is not included into comparison because it was too slow to compete.

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-03:

#16

Agreed, uncompressed tar doesn't stand a chance against anything you tested.

try

(donor) tar | zstd -T4 | socat socat | zstd -d -T4 ] tar (joiner)

.. with zstd >= 1.2.0 :)

Speedup and size reduction impressive, and our bottleneck is network IO, not disk.

xbstream already does some compression (albeit less efficient than zstd), adding another compression layer would provide some (if not identical) size gain as with tar streaming, but cost additional CPU cycles we avoid by not using xbstream, thus avoiding its compression.

Revision history for this message

Sergei Glushchenko (sergei.glushchenko) wrote on 2017-08-03:

#17

xbstream does not compress anything unless you tell it to.

xtrabackup --backup --stream=xbstream > /dev/null

is faster than

xtrabackup --backup --stream=tar > /dev/null

It seems that libarchive which we using is not very efficient.

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-03:

#18

Hmm. There might have been another reason - if I remember correctly, you can stream xbstream for backup, but not for restore, while you can stream the tar. You'd need the unpackaged xbstream version on the same filesystem for the restore, which isn't always feasible with multi-Terabyte backups.

Thus, I'd love to have the tar version fixed, please :)

Revision history for this message

Vasily Nemkov (vasily.nemkov) wrote on 2017-08-03:

#19

Only 2.4 affected

Revision history for this message

Vasily Nemkov (vasily.nemkov) wrote on 2017-08-03:

#20

PR: https://github.com/percona/percona-xtrabackup/pull/408

Revision history for this message

Vasily Nemkov (vasily.nemkov) wrote on 2017-08-10:

#21

Merged PR into 2.4

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-08-21:

#22

Thanks :) is there a planned release date for 2.4.9?

Revision history for this message

Andreas 'count' Kotes (berlincount) wrote on 2017-10-10:

#23

Hello :) I agree that there's a fix going into 2.4.10 - but .. when will the fix not only be "released", but available as a package to customers? We'll have to put code to automatically downgrade to 2.4.7 into our instrumentation otherwise :(

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2018-01-20:

#24

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-507

	Status	Importance	Assigned to	Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB	Status tracked in 2.4
2.2	Invalid	Undecided	Unassigned
2.3	Invalid	Undecided	Unassigned
2.4	Fix Released	High	Vasily Nemkov	Percona XtraBackup moved to https://jira.percona.com/projects/PXB 2.4.9

Percona XtraBackup moved to https://jira.percona.com/projects/PXB

using innobackupx with --slave-info output file xtrabackup_slave_info does not contain slave information in 2.4.8[]

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches