Database page corruption detected at page 1 with encryption enabled table full backup

[Shahriyar Rzayev]

I have a sample table with encryption enabled:

CREATE TABLE `sbtest1` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `k` int(10) unsigned NOT NULL DEFAULT '0',
  `c` char(120) NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  KEY `k_1` (`k`)
) ENGINE=InnoDB AUTO_INCREMENT=200001 DEFAULT CHARSET=latin1 MAX_ROWS=1000000 ENCRYPTION='Y'

And an infinite loop:

[root@ps-pxb-5 ~]# cat alter_rotate_keyring.py
import mysql.connector

cnx = mysql.connector.connect(user='root', password='Baku12345#',
                              host='127.0.0.1',
                              database='dbtest')
cursor = cnx.cursor()

alter_stmt = "ALTER INSTANCE ROTATE INNODB MASTER KEY"

while(True):
 print "Master key rotated"
 cursor.execute(alter_stmt)

cursor.close()
cnx.close()

Steps:
1. Run python script (altering master key) -> leave running
2. alter table sbtest1 compression='zlib';
3. /usr/local/xtrabackup/bin/xtrabackup --defaults-file=/etc/my.cnf --backup --keyring-file-data=/var/lib/mysql-keyring/keyring --server-id=0 --datadir=/var/lib/mysql/ --target-dir=/home/backup_dir/full/ --user=root --password=Baku12345# --no-version-check

Result:

160516 12:37:31 Connecting to MySQL server host: localhost, user: root, password: set, port: 0, socket: /var/lib/mysql/mysql.sock
Using server version 5.7.11-4
/usr/local/xtrabackup/bin/xtrabackup version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64) (revision id: b43f6fb)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql/
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
InnoDB: Number of pools: 1
160516 12:37:31 >> log scanned up to (3218914516)
xtrabackup: Generating a list of tablespaces
InnoDB: Allocated tablespace ID 35 for mysql/plugin, old maximum was 0
160516 12:37:32 [01] Copying ./ibdata1 to /home/backup_dir/full/ibdata1
160516 12:37:33 >> log scanned up to (3218919028)
160516 12:37:34 >> log scanned up to (3218922543)
160516 12:37:34 [01] ...done
160516 12:37:50 [01] Copying ./db2/sbtest1.ibd to /home/backup_dir/full/db2/sbtest1.ibd
160516 12:37:51 >> log scanned up to (3218987189)
160516 12:37:52 >> log scanned up to (3218990704)
160516 12:37:53 >> log scanned up to (3218994310)
160516 12:37:54 >> log scanned up to (3218997825)
160516 12:37:55 >> log scanned up to (3219001431)
160516 12:37:56 >> log scanned up to (3219004894)
160516 12:37:57 >> log scanned up to (3219008607)
160516 12:37:58 >> log scanned up to (3219012304)
160516 12:37:59 >> log scanned up to (3219016290)
160516 12:38:00 >> log scanned up to (3219019896)
160516 12:38:01 >> log scanned up to (3219023775)
160516 12:38:02 >> log scanned up to (3219027488)
160516 12:38:03 [01] ...done
160516 12:38:03 [01] Copying ./dbtest/sbtest1.ibd to /home/backup_dir/full/dbtest/sbtest1.ibd
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
160516 12:38:03 >> log scanned up to (3219030896)
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Error: failed to read page after 10 retries. File ./dbtest/sbtest1.ibd seems to be corrupted.
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.

[Shahriyar Rzayev (rzayev-sehriyar)]

Same thing if you try to backup while doing "alter table sbtest1 compression="

160523 10:12:41 [01] ...done
160523 10:12:41 [01] Copying ./dbtest/#sql-1665_3.ibd to /home/backup_dir/full/dbtest/#sql-1665_3.ibd
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
160523 10:12:41 >> log scanned up to (35313522605)
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Database page corruption detected at page 1, retrying...
[01] xtrabackup: Error: failed to read page after 10 retries. File ./dbtest/#sql-1665_3.ibd seems to be corrupted.
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/211

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-477