Page directory corruption during xtrabackup --apply-log recovery mysql 5.5. with xtrabackup-1.6

[Sanjeev Sagar]

xtrabackup recovery issue with mysql5.5. xtrabackup-1.6 on 64 bit

mysql/xtrabackup/bin/innobackupex-1.5.1 --apply-log --ibbackup=/mysql/db01/xtrabackup-1.6/bin/xtrabackup_55 --defaults-file=/usr/local/mysql/data/my.cnf /usr/local/mysql/data/

In the end of the recovery it is failing with the following error and it gives the page ascii dump.

InnoDB: Doing recovery: scanned up to log sequence number 1117752400325 (100 %)
110920 2:25:02 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 65503647, file name /mysql/binlogs/log-bin.000840
InnoDB: Page directory corruption: infimum not pointed to
110920 2:25:43 InnoDB: Page dump in ascii and hex (16384 bytes):

[Valentine Gostev (longbow)]

Sanjeev,
can you please post the content of my.cnf file?
Also OS version and server version this data files were created with might be useful.

Thanks!

[Sanjeev Sagar (sanjeev-sagar)]

Hello, Please see below for the requested info:

uname -a
Linux 2.6.18-238.5.1.el5 #1 SMP Mon Feb 21 05:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release
Red Hat Enterprise Linux Client release 5.6 (Tikanga)

mysql release: mysql-5.5.13-linux2.6-x86_64

{/usr/local/mysql/data}>cat my.cnf
[mysqld]
# skip-slave-start
skip-external-locking
log-slave-updates
log-slow-slave-statement
slow_query_log

port = 3306
socket = /tmp/mysql.sock
user = mysql
key_buffer_size = 300M
back_log = 50
max_connections = 300
max_connect_errors = 50
table_open_cache = 2048
max_allowed_packet = 200M
binlog_cache_size = 1M
max_heap_table_size = 64M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
thread_cache_size = 8
thread_concurrency = 8
query_cache_size = 64M
query_cache_limit = 2M
ft_min_word_len = 4
thread_stack = 192K
transaction_isolation = REPEATABLE-READ
tmpdir = /mysql/db01/tmp
tmp_table_size = 64M
open-files-limit = 32000
bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1
myisam_recover

log-error = /usr/local/mysql/data/s4r2-error.err
log-bin-index = /mysql/binlogs/s4r2-log-bin.index
log-bin = /mysql/binlogs/s4r2-log-bin
binlog_format = mixed
max_binlog_size = 250M
slow-query-log-file = /mysql/binlogs/slow-queries.log
long_query_time = 2
server-id = 1100402

# Innodb Plugin 1.1 variables

innodb_buffer_pool_size = 4G # Optimal number as per testing for 2/3/4/6 G
innodb_additional_mem_pool_size = 16M
innodb_data_home_dir = /mysql/db01/ibdata
innodb_log_group_home_dir = /mysql/binlogs/iblogs
innodb_data_file_path = ibdata1:1024M;ibdata2:1024M;ibdata3:1024M:autoextend
innodb_log_files_in_group = 3
innodb_log_file_size = 1024M
innodb_log_buffer_size = 8M
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_flush_log_at_trx_commit = 2
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 120

default-storage-engine = Innodb
innodb_file_per_table = 1
innodb_file_format = barracuda
innodb_strict_mode = 1

innodb_fast_shutdown = 0 # slow shutdown

# Defaults in mysql 5.5 for Innodb plugin

#ignore_builtin_innodb ...

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Sanjeev Sagar (sanjeev-sagar)]

Hello Stewart, Is there any update on this Xtrabackup issue? or any work around? It's creating a big problem for me while restoring the xtrabackup.

I'll highly appreciate if you could let me know.

Regards,

[Stewart Smith (stewart)]

The more information you can provide us to help reproduce it the better. With the current information here we would have to spend a decent amount of time reproducing. Can you try and provide a set of steps going from fresh install to failed --apply-logs ?

(there is also the option of engaging Percona to look at it for you - including existing setup, in which case it moves very much to the top of the priority list :)

[Sanjeev Sagar (sanjeev-sagar)]

Well... I've already provided the my.cnf file and system info.

  It's a remote recovery, so following are more steps

1. ssh <machine> "cat <xtrabackup-file-name>" |tar -ixvf -

2. On the destination:

/mysql/xtrabackup/bin/innobackupex-1.5.1 --apply-log --ibbackup=/mysql/xtrabackup/bin/xtrabackup_55 --defaults-file=/usr/local/mysql/data/my.cnf /usr/local/mysql/data/

3. Move the ibdata files and binlogs files to /mysql/binlogs and /mysql/db01/ibdata location

4. Start the db.

Let me know in case of any further questions.

Regards

On 02/13/2012 04:53 PM, Stewart Smith wrote:
> The more information you can provide us to help reproduce it the better.
> With the current information here we would have to spend a decent amount
> of time reproducing. Can you try and provide a set of steps going from
> fresh install to failed --apply-logs ?
>
> (there is also the option of engaging Percona to look at it for you -
> including existing setup, in which case it moves very much to the top
> of the priority list :)
>

[Alexey Kopytov (akopytov)]

Is this bug still repeatable with with the latest XtraBackup version?

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Matthijs Mohlmann (matthijs)]

Well, I got the same problem as described, these are my steps to reproduce:

Step 1:
/usr/bin/innobackupex --no-timestamp --compact --compress --compress-threads=4 --parallel=4 --defaults-extra-file=/var/tmp/client.cnf --include='^dbname[.]*' --stream=xbstream | xbstream -x -C <some directory>

Step 2:
/usr/bin/innobackupex --decompress --decompress --parallel=4 <some directory>

Step 3:
/usr/bin/innobackupex --apply-log --redo-only <some directory>

Step 4 (step that fails):
/usr/bin/innobackupex --apply-log --export

I've tried to do Step 3 and Step 4 at once, but I also need to apply incrementals if there are any.

This process completes succesfully when I remove the --compact option.

If you need more information, do not hesitate to ask.

Regards, Matthijs

[Matthijs Mohlmann (matthijs)]

Seems that I cannot edit my comment:

Of course <some directory> is also added at step 4:

Step 4:
/usr/bin/innobackupex --apply-log --export <some directory>

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Matthijs

For compact, I have reported a separate bug - https://bugs.launchpad.net/percona-xtrabackup/+bug/1192834 You can provide details there.

[Alexey Kopytov (akopytov)]

Yes, the problem with --compact is tracked in bug #1192834. Closing this report as invalid.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-311