Include innodb_log_group_home_dir and innodb_undo_directory as special variables supported by sst_special_dirs

Bug #1382347 reported by Jaime Sicam on 2014-10-17
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Medium
Sergei Glushchenko
2.2
Fix Released
Medium
Sergei Glushchenko
2.3
Fix Released
Medium
Sergei Glushchenko

Bug Description

Currently, xtrabackup-v2 fails when innodb_log_group_home_dir or innodb_undo_directory is set

Tested on PXC 5.6.20/CentOS 6:

Joiner my.cnf:
[mysqld]
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://192.168.1.80,192.168.1.81,192.168.1.82

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #2 address
wsrep_node_address=192.168.1.81

# SST method
wsrep_sst_method=xtrabackup-v2

# Cluster name
wsrep_cluster_name=my_centos_cluster

# Authentication for SST method
wsrep_sst_auth="sstuser:s3cret"

datadir=/var/lib/mysql
innodb_data_home_dir=/var/lib/mysql-innodb
innodb_log_group_home_dir=/var/lib/mysql-innodb/logs
#innodb_undo_directory=/var/lib/mysql-innodb/logs

innodb_file_per_table
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[sst]
sst_special_dirs=1

Donor my.cnf:
[mysqld]
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://192.168.1.80,192.168.1.81,192.168.1.82

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #3 address
wsrep_node_address=192.168.1.82

# SST method
wsrep_sst_method=xtrabackup-v2

# Cluster name
wsrep_cluster_name=my_centos_cluster

# Authentication for SST method
wsrep_sst_auth="sstuser:s3cret"

datadir=/var/lib/mysql
innodb_data_home_dir=/var/lib/mysql-innodb
innodb_log_group_home_dir=/var/lib/mysql-innodb/logs
#innodb_undo_directory=/var/lib/mysql-innodb/logs

innodb_file_per_table

[sst]
sst_special_dirs=1

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Test:
[root@pxc2 mysql]# service mysql restart
Shutting down MySQL (Percona XtraDB Cluster) ERROR! MySQL (Percona XtraDB Cluster) PID file could not be found!
 ERROR! MySQL (Percona XtraDB Cluster) is not running, but lock file (/var/lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)......State transfer in progress, setting sleep higher
......... ERROR! The server quit without updating PID file (/var/lib/mysql/pxc2.pid).
 ERROR! MySQL (Percona XtraDB Cluster) server startup failed!
 ERROR! Failed to restart server.

Logs:
cat /var/lib/mysql/.sst/innobackup.move.log

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

IMPORTANT: Please check that the move-back run completes successfully.
           At the end of a successful move-back run innobackupex
           prints "completed OK!".

innobackupex: Error: Original InnoDB log directory '/var/lib/mysql-innodb/logs' does not exist! at /usr//bin/innobackupex line 2141.

Error log:
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20141016 12:31:53.054)
removed directory: `/var/lib/mysql-innodb/logs'
removed `/var/lib/mysql/gvwstate.dat'
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20141016 12:31:53.137)
2014-10-16 12:32:36 16914 [Note] WSREP: 0.0 (pxc3): State transfer to 1.0 (pxc2) complete.
2014-10-16 12:32:36 16914 [Note] WSREP: Member 0.0 (pxc3) synced with group.
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20141016 12:32:47.878)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20141016 12:32:47.897)
rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or directory
rm: cannot remove `/var/lib/mysql//innobackup.move.log': No such file or directory
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20141016 12:33:03.580)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20141016 12:33:03.594)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20141016 12:33:05.472)
WSREP_SST: [INFO] Removing the sst_in_progress file (20141016 12:33:05.475)
2014-10-16 12:33:05 16914 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.1.81' --auth 'sstuser:s3cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '16914' '' : 1 (Operation not permitted)
2014-10-16 12:33:05 16914 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-10-16 12:33:05 16914 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2014-10-16 12:33:05 16914 [ERROR] Aborting

Przemek (pmalkowski) wrote :
Download full text (3.8 KiB)

I confirm this problem for PXC 5.6.20. When the innodb_log_group_home_dir points to a directory inside innodb_data_home_dir - xtrabackup-v2 SST method fails.

percona1 mysql> select @@version,@@version_comment;
+----------------+---------------------------------------------------------------------------------------------------+
| @@version | @@version_comment |
+----------------+---------------------------------------------------------------------------------------------------+
| 5.6.20-68.0-56 | Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126 |
+----------------+---------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

percona1 mysql> show variables like 'innodb_%dir%';
+--------------------------------+-------------------+
| Variable_name | Value |
+--------------------------------+-------------------+
| innodb_data_home_dir | /data/innodb |
| innodb_log_arch_dir | /data/innodb/logs |
| innodb_log_group_home_dir | /data/innodb/logs |
| innodb_max_dirty_pages_pct | 75 |
| innodb_max_dirty_pages_pct_lwm | 0 |
| innodb_undo_directory | /data/innodb |
+--------------------------------+-------------------+
6 rows in set (0.00 sec)

in my.cnf:
[sst]
streamfmt=xbstream
sst_special_dirs=1

Forced SST to percona1 node:

WSREP_SST: [INFO] Proceeding with SST (20141018 10:58:14.247)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log directories (20141018 10:58:14.250)
removed `/data/innodb/ibdata1'
removed `/data/innodb/logs/ib_logfile1'
removed `/data/innodb/logs/ib_logfile0'
removed directory: `/data/innodb/logs'
removed `/data/innodb/undo002'
removed `/data/innodb/undo001'
(...)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20141018 10:58:14.295)
2014-10-18 10:58:29 18105 [Note] WSREP: 1.0 (percona2): State transfer to 2.0 (percona1) complete.
2014-10-18 10:58:29 18105 [Note] WSREP: Member 1.0 (percona2) synced with group.
WSREP_SST: [INFO] Compressed qpress files found (20141018 10:58:29.062)
WSREP_SST: [INFO] Decompression with 2 threads (20141018 10:58:29.067)
WSREP_SST: [INFO] Evaluating find /data/myisam//.sst -type f -name '*.qp' -printf '%p\n%h\n' | xargs -n 2 qpress -T2d (20141018 10:58:29.071)
WSREP_SST: [INFO] Removing qpress files after decompression (20141018 10:58:29.646)
WSREP_SST: [INFO] Preparing the backup at /data/myisam//.sst (20141018 10:58:29.663)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20141018 10:58:29.666)
WSREP_SST: [INFO] Moving the backup to /data/myisam/ (20141018 10:58:34.923)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20141018 10:58:34.927)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20141018 10:58:35.524)
WSREP...

Read more...

Changed in percona-xtradb-cluster:
status: New → Confirmed
Przemek (pmalkowski) wrote :

Confirmed also for:
mysql> select @@version,@@version_comment;
+----------------+-----------------------------------------------------------------------------------------------------+
| @@version | @@version_comment |
+----------------+-----------------------------------------------------------------------------------------------------+
| 5.5.39-36.0-55 | Percona XtraDB Cluster (GPL), Release rel36.0, Revision 824, WSREP version 25.11, wsrep_25.11.r4023 |
+----------------+-----------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

The sst_special_dirs setting does not help.

This is a bug in innobackupex, when it does --move, it has to create the entire treewith install, this is especially true if the tree of datadir and other special directories are not same wrt. directories.

Alexey Kopytov (akopytov) wrote :

innobackupex should make sure that all target directories for --copy-back/--move-back:

1) exist and empty (this is what we do now)
2) or, do not exist, in which case they should be created with mkpath().

no longer affects: percona-xtradb-cluster
no longer affects: percona-xtradb-cluster/5.5
no longer affects: percona-xtradb-cluster/5.6

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-710

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers