Prepare error: Tablespace size stored in header is X pages, but the sum of data file sizes is only Y pages

[Shahriyar Rzayev]

Here are full steps:

CREATE TABLE `sbtest1` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `k` int(10) unsigned NOT NULL DEFAULT '0',
  `c` char(120) NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  KEY `k_1` (`k`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Running:

sysbench --test=/usr/share/doc/sysbench/tests/db/insert.lua --oltp-test-mode=complex --num-threads=200 --mysql-db=dbtest --mysql-user=root --mysql-password=Baku12345# --db-driver=mysql --max-requests=0 run

Taking backups:

xtrabackup --defaults-file=/etc/my.cnf --backup --datadir=/var/lib/mysql/ --target-dir=/home/backup_dir/full/ --user=root --password=Baku12345# --no-version-check

xtrabackup --defaults-file=/etc/my.cnf --backup --target-dir=/home/backup_dir/inc/inc1 --incremental-basedir=/home/backup_dir/full --datadir=/var/lib/mysql/ --user=root --password=Baku12345# --no-version-check

xtrabackup --defaults-file=/etc/my.cnf --backup --target-dir=/home/backup_dir/inc/inc2 --incremental-basedir=/home/backup_dir/inc/inc1 --datadir=/var/lib/mysql/ --user=root --password=Baku12345# --no-version-check

xtrabackup --defaults-file=/etc/my.cnf --backup --target-dir=/home/backup_dir/inc/inc3 --incremental-basedir=/home/backup_dir/inc/inc2 --datadir=/var/lib/mysql/ --user=root --password=Baku12345# --no-version-check

Preparing backups:

xtrabackup --defaults-file=/home/backup_dir/full/backup-my.cnf --prepare --apply-log-only --target-dir=/home/backup_dir/full

xtrabackup --defaults-file=/home/backup_dir/full/backup-my.cnf --prepare --apply-log-only --target-dir=/home/backup_dir/full --incremental-dir=/home/backup_dir/inc/inc1

xtrabackup --defaults-file=/home/backup_dir/full/backup-my.cnf --prepare --apply-log-only --target-dir=/home/backup_dir/full --incremental-dir=/home/backup_dir/inc/inc2

[root@centos7-master inc]# xtrabackup --defaults-file=/home/backup_dir/full/backup-my.cnf --prepare --target-dir=/home/backup_dir/full --incremental-dir=/home/backup_dir/inc/inc3
xtrabackup version 2.4.1 based on MySQL server 5.7.10 Linux (x86_64) (revision id: a2dc9d4)
incremental backup from 1052459319 is enabled.
xtrabackup: cd to /home/backup_dir/full
xtrabackup: This target seems to be already prepared with --apply-log-only.
InnoDB: Number of pools: 1
xtrabackup: xtrabackup_logfile detected: size=11272192, start_lsn=(1156966840)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /home/backup_dir/inc/inc3
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 11272192
xtrabackup: Generating a list of tablespaces
xtrabackup: page size for /home/backup_dir/inc/inc3/ibdata1.delta is 16384 bytes
Applying /home/backup_dir/inc/inc3/ibdata1.delta to ./ibdata1...
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /home/backup_dir/inc/inc3
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 11272192
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Number of pools: 1
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -20
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 1156966840
InnoDB: Doing recovery: scanned up to log sequence number 1162209280 (52%)
InnoDB: Doing recovery: scanned up to log sequence number 1166956838 (99%)
InnoDB: Doing recovery: scanned up to log sequence number 1157425152 (4%)
InnoDB: Doing recovery: scanned up to log sequence number 1162668032 (56%)
InnoDB: Doing recovery: scanned up to log sequence number 1166956838 (99%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Doing recovery: scanned up to log sequence number 1157883904 (9%)
InnoDB: Doing recovery: scanned up to log sequence number 1163126784 (61%)
InnoDB: Doing recovery: scanned up to log sequence number 1166956838 (99%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Creating shared tablespace for temporary tables
InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
InnoDB: File './ibtmp1' size is now 12 MB.
InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active.
InnoDB: 32 non-redo rollback segment(s) are active.
InnoDB: Waiting for purge to start
InnoDB: Tablespace size stored in header is 64128 pages, but the sum of data file sizes is only 62656 pages
InnoDB: Cannot start InnoDB. The tail of the system tablespace is missing. Have you edited innodb_data_file_path in my.cnf in an InnoDB: inappropriate way, removing ibdata files from there? You can set innodb_force_recovery=1 in my.cnf to force InnoDB: a startup if you are trying to recover a badly corrupt database.
InnoDB: Plugin initialization aborted with error Generic error
xtrabackup: innodb_init(): Error occured.

[Sergei Glushchenko (sergei.glushchenko)]

Able to consistently reproduce on my laptop using following test case.

MYSQLD_EXTRA_MY_CNF_OPTS="
innodb-log-file-size=60M
innodb-log-files-in-group=4
innodb-file-per-table=0
innodb-buffer-pool-size=2G
"

start_server

$MYSQL $MYSQL_ARGS test <<EOF &

create table t (a INT) engine=InnoDB;

insert into t values (1), (2), (3), (4);

start transaction;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
commit;
start transaction;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
commit;
start transaction;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
insert into t select * from t;
rollback;
EOF

sleep 120

$MYSQL $MYSQL_ARGS -e 'show processlist'

xtrabackup --backup --target-dir=$topdir/bac1
xtrabackup --prepare --target-dir=$topdir/bac1

wait

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/236

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-462