Comment 13 for bug 1192834

Any bumps needed here or is a solution in sight?

This was throwing this in syslog:

Nov 24 07:19:25 db02 mysqld: WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20141124 07:19:25.536)
Nov 24 07:19:25 db02 mysqld: WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20141124 07:19:25.547)
Nov 24 07:19:37 db02 mysqld: WSREP_SST: [ERROR] Cleanup after exit with status:1 (20141124 07:19:37.391)
Nov 24 07:19:37 db02 mysqld: WSREP_SST: [INFO] Removing the sst_in_progress file (20141124 07:19:37.407)
Nov 24 07:19:37 db02 mysqld: 2014-11-24 07:19:37 9103 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.100.101.23' --auth 'wsrep_user:' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '9103' '' : 1 (Operation not permitted)
Nov 24 07:19:37 db02 mysqld: 2014-11-24 07:19:37 9103 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
Nov 24 07:19:37 db02 mysqld: 2014-11-24 07:19:37 9103 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
Nov 24 07:19:37 db02 mysqld: 2014-11-24 07:19:37 9103 [ERROR] Aborting

prepare log would end with something like this but with a lot more 0s

Expanding ./mysql/innodb_table_stats.ibd
Expanding ./mysql/innodb_index_stats.ibd
Expanding ./mysql/slave_master_info.ibd
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Not using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: The log sequence numbers 882919419 and 882919419 in ibdata files do not match the log sequence number 888968847 in the ib_logfiles!
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 12199785, file name mysql-bin.000004
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
InnoDB: Page directory corruption: infimum not pointed to
2014-11-24 07:58:12 7f5d6d7a6700 InnoDB: Page dump in ascii and hex (16384 bytes):
len 16384; hex deadbeef00000000000000000000000000000000000000000000000000000000000000000000434f4d504143545000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

Something that I did soon before this all went down, which may or may not be relevant, is that I imported a database into percona with some tables using myisam. I then converted them while they were running live into innodb:

# for t in `echo "show tables" | mysql --batch --skip-column-names dbname`; do mysql dbname -e "ALTER TABLE \`$t\` ENGINE = InnoDB;"; done

cheers