Apply fails for compressed pages without data

[Daniël van Eeden]

With Xtrabackup 2.4.8 take a backup of MySQL 5.7.20 with innodb_log_compressed_pages=OFF.
This succeeds.

With Xtrabackup 2.4.8 restore the backup.
This fails.

====================================================================================
InnoDB: Doing recovery: scanned up to log sequence number 59571174338048 (99%)
InnoDB: Doing recovery: scanned up to log sequence number 59571176730860 (99%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
InnoDB: Trx id counter is 12727518208
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2017-12-02 16:52:52 0x7f34d448d700 InnoDB: Assertion failure in thread 139864876570368 in file page0zip.ic line 419
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
15:52:52 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2c)[0xcef4ac]
xtrabackup(handle_fatal_signal+0x262)[0xa06812]
/lib64/libpthread.so.0(+0xf5e0)[0x7f4ea976d5e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f4ea73f01f7]
/lib64/libc.so.6(abort+0x148)[0x7f4ea73f18e8]
xtrabackup[0x6f4849]
xtrabackup[0x80fb61]
xtrabackup(_Z22recv_recover_page_funcmP11buf_block_t+0xacc)[0x810cbc]
xtrabackup(_Z20buf_page_io_completeP10buf_page_tb+0x485)[0x8f0225]
xtrabackup(_Z12fil_aio_waitm+0x12f)[0x8b5d0f]
xtrabackup(io_handler_thread+0x28)[0x7e5788]
/lib64/libpthread.so.0(+0x7e25)[0x7f4ea9765e25]
/lib64/libc.so.6(clone+0x6d)[0x7f4ea74b334d]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
====================================================================================

[Sergei Glushchenko (sergei.glushchenko)]

It is known issue that innodb_log_compressed_pages=OFF doesn't work well with PXB. Different versions of zlib can produce compressed pages of different length on the same data which leads to PXB needing a page split when MySQL does not need it. Which lead to this or similar assert. There are two options:

1. innodb_log_compressed_pages=ON
2. Make sure PXB and MySQL are built with exactly same zlib version. We build PS and PXB packages with system zlib. MySQL community comes with bundled zlib. By making sever and PXB to use the same version you can ensure that zlib compress will produce the same result.

[Daniël van Eeden (dveeden)]

$ strings $(which xtrabackup) | grep 'Compressed tables use zlib'
Compressed tables use zlib 1.2.7
$ strings $(which mysqld) | grep 'Compressed tables use zlib'
Compressed tables use zlib 1.2.3

I indeed don't expect XtraBackup to work with different zlib versions.
However I do think it should print a warning when making the backup.

[Sergei Glushchenko (sergei.glushchenko)]

Good point. I see couple of issues here:

1. there seems to be no good way to find out which zlib version is in use by server (except maybe parsing an mysql.err which is not always possible)
2. prepare (this error pops out only during prepare stage) can be done on different host with different environment, it means we need to save zlib version along with the backup and abort early if it doesn't match

[Daniël van Eeden (dveeden)]

1. I wrote a patch to make that easier
https://github.com/mysql/mysql-server/pull/184

2. That would be a nice, but complex solution.
Also then the warning/failure is during restore instead of during backup.
I like warnings/failures to happen during backup as that's easier to fix.
During restore I'm usually more stressed and under time pressure.

What about a simple solution:
If innodb_log_compressed_pages=OFF when making the backup then print
a warning saying that this is not recommended.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1485