xtrabackup incremental fails in 5.7 due to wrong changed pages bitmap info

[Rick Pizzi]

xtrabackup version 2.4.8 based on MySQL server 5.7.13 Linux (x86_64) (revision id: 97330f7)
MySQL Server version: 5.7.18-15.1

When trying to perform an incremental backup that uses the changed pages bitmap it always fails with the following error:

170802 15:54:11 [16] Streaming ./abc/CTRL_DM_ANOMALY_TEMP.ibd
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: 262144 bytes should have been read. Only 147456 bytes read. Retrying for the remaining bytes.
InnoDB: Retry attempts for reading partial data failed.
InnoDB: Tried to read 262144 bytes at offset 0, but was only able to read 147456
InnoDB: File (unknown): 'read' returned OS error 0. Cannot continue operation
InnoDB: Cannot continue operation.

Looks like this happens with tables which get changed frequently.

I did a bit of investigation in the code and looks like the information from the bitmap is wrong, as it reports size for some of the tablespace which is greater than the actual size on disk and therefore xtrabackup fails because it is trying to read past end of file in get_next_batch().

If we run the incremental backup without using the changed pages bitmap it works fine.
Full backup works fine too, for the very same reason of not using the bitmap.

[Rick Pizzi (pizzi)]

I am able to reproduce this on one of our slaves (not on the others), MySQL version and Xtrabackup version being the same.

Apparently it is due, at least in this case, to a mix of insert and truncate table instructions.
I believe the truncate table is the key to the bug here....

Just create the schema, run the attached script on the master, and run an xtrabackup incremental on the slave; hopefully it will break on that table with error posted above.

We run the incremental as follows:

/usr/bin/innobackupex --defaults-file=/etc/my.cnf --no-version-check --incremental --no-backup-locks --slave-info --no-timestamp --parallel=1 --socket=/db/data/mysql.sock --user=backup --password=amended --tmpdir=/storage/backup/tmp --extra-lsndir=/storage/backup/lsn_incr --incremental-basedir=/storage/backup/lsn_incr /storage/backup/incr

Versions:

mysqld Ver 5.7.19-17 for Linux on x86_64 (Percona Server (GPL), Release 17, Revision e19a6b7b73f)
xtrabackup version 2.4.8 based on MySQL server 5.7.13 Linux (x86_64) (revision id: 97330f7)

[Rick Pizzi (pizzi)]

Please disregard this bug report and see https://bugs.launchpad.net/percona-xtrabackup/+bug/1731260 instead, which has full information about how to reproduce the issue. Thanks and sorry for the dupe.

[Sergei Glushchenko (sergei.glushchenko)]

I will set this one as duplicate for bug 1731260 because later one has clear reproduction steps.