Be able to follow symlinks in the restore process

[Peiran]

Our innodb files do not fit in one disk partition and are spread on two partitions:

/dev/sdc1 /mysql/data
/dev/sdd1 /var/lib/mysql/

During restore process, we tried to use symlinks but the restore process failed with assertion failure:

# ls -l /mysql/data
lrwxrwxrwx 1 mysql mysql 26 Jun 30 11:49 acct1 -> /var/lib/mysql/data/acct1/
drwx------ 2 mysql mysql 4096 Jun 25 21:56 acct2
drwx------ 2 mysql mysql 4096 Jun 25 21:56 acct3
drwx------ 2 mysql mysql 4096 Jun 25 21:57 acct4
lrwxrwxrwx 1 mysql mysql 29 Jun 30 11:49 ibdata1 -> /var/lib/mysql/ibdata/ibdata1
....
-rw-r----- 1 mysql mysql 265 Jun 25 20:52 backup-my.cnf
-rw-r----- 1 mysql mysql 29 Jun 25 21:57 xtrabackup_binlog_info
-rw-r----- 1 mysql mysql 101 Jun 25 21:57 xtrabackup_checkpoints
-rw-r----- 1 mysql mysql 600 Jun 25 21:57 xtrabackup_info
-rw-rw---- 1 dba dba 2962243584 Jun 25 21:57 xtrabackup_logfile

# innobackupex --use-memory=20G --apply-log /mysql/data
....
InnoDB: Doing recovery: scanned up to log sequence number 4012072696832 (99%)
InnoDB: Doing recovery: scanned up to log sequence number 4012077939712 (99%)
InnoDB: Doing recovery: scanned up to log sequence number 4012080156840 (99%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2015-06-30 11:54:17 7f6ff844a700 InnoDB: Assertion failure in thread 140118883346176 in file log0recv.cc line 1280
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
0 2015-06-30 11:54:17 7f6ff8e4b700 InnoDB: Assertion failure in thread 140118893836032 in file log0recv.cc line 1280
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2649
        main::apply_log() called at /usr/bin/innobackupex line 1578
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2649.

[Nilnandan Joshi (nilnandan-joshi)]

Hi,

I tried to reproduce but got something different error.

root@desktop:/home/nilnandan/backup/2015-07-07_12-05-41# ll
total 92
drwxr-xr-x 5 root root 4096 Jul 7 12:10 ./
drwxrwxr-x 3 nilnandan nilnandan 4096 Jul 7 12:05 ../
-rw-r--r-- 1 root root 358 Jul 7 12:05 backup-my.cnf
lrwxrwxrwx 1 root root 23 Jul 7 12:10 ibdata1 -> /home/nilnandan/ibdata1
drwx------ 2 root root 16384 Jul 7 12:05 mysql/
drwxr-xr-x 2 root root 12288 Jul 7 12:05 performance_schema/
drwxr-xr-x 2 root root 4096 Jul 7 12:05 test/
-rw-r----- 1 root root 95 Jul 7 12:05 xtrabackup_checkpoints
-rw-r--r-- 1 root root 530 Jul 7 12:05 xtrabackup_info
-rw-r----- 1 root root 2560 Jul 7 12:05 xtrabackup_logfile
root@desktop:/home/nilnandan/backup/2015-07-07_12-05-41#
root@desktop:/home/nilnandan/backup/2015-07-07_12-05-41#

root@desktop:/home/nilnandan/backup# innobackupex --apply-log 2015-07-07_12-05-41/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

...

2015-07-07 12:10:34 7fc0ecc01700 InnoDB: Error: page 304 log sequence number 8005953003
InnoDB: is in the future! Current system log sequence number 8005938198.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: for more information.
InnoDB: 5.6.24 started; log sequence number 8005938188

[notice (again)]
  If you use binary log and don't use any hack of group commit,
  the binary log position seems to be:

xtrabackup: starting shutdown with innodb_fast_shutdown = 1
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 8005959116
150707 12:10:35 innobackupex: completed OK!
root@desktop:/home/nilnandan/backup# ^C
root@desktop:/home/nilnandan/backup#

Can you please provide the exact command that how you take the backup and restore it? also /mysql/data is also the backup dir? I'm bit confused because you are running innobackupex --use-memory=20G --apply-log /mysql/data .

Please also provide the output of data dir content.

[Peiran (peiran-song-5)]

Hi Nil,

The backup command is not fancy:

ssh dbhost “innobackupex --user=root --password=secret --slave-info --safe-slave-backup --stream=xbstream --parallel=4 /tmp 2>/dblog/innobackupex.log”|xbstream -x -C /backupdir

/mysql/data is the datadir where we run the --apply-log.

The first time I tried this, the restore failed twice when using symlinks, but the same backup restored successfully without symlinks.

Good news is as I just tried the restore with symlinks again, it worked! A couple backup both restored successfully with symlinks.

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1335