Getting InnoDB internal dictionary errors from joiner after receiving a successful rsync SST when special directories are used

[Jaime Sicam]

Tested on PXC 5.6.20/CentOS 6:
- special directory used is innodb_data_home_dir
- if there's a plan to fix this, please also consider variables innodb_log_group_home_dir and innodb_undo_directory to be fixed as well

Joiner my.cnf:
[mysqld]
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://192.168.1.80,192.168.1.81,192.168.1.82

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #2 address
wsrep_node_address=192.168.1.81

# SST method
wsrep_sst_method=rsync

# Cluster name
wsrep_cluster_name=my_centos_cluster

# Authentication for SST method
wsrep_sst_auth="sstuser:s3cret"

datadir=/var/lib/mysql
innodb_data_home_dir=/var/lib/mysql-innodb

innodb_file_per_table
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[sst]
sst_special_dirs=1

Donor my.cnf:
[mysqld]
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

user=mysql

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://192.168.1.80,192.168.1.81,192.168.1.82

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #3 address
wsrep_node_address=192.168.1.82

# SST method
wsrep_sst_method=rsync

# Cluster name
wsrep_cluster_name=my_centos_cluster

# Authentication for SST method
wsrep_sst_auth="sstuser:s3cret"

datadir=/var/lib/mysql
innodb_data_home_dir=/var/lib/mysql-innodb

innodb_file_per_table

[sst]
sst_special_dirs=1

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Test:
[root@pxc2 ~]# service mysql restart
Shutting down MySQL (Percona XtraDB Cluster) ERROR! MySQL (Percona XtraDB Cluster) PID file could not be found!
Starting MySQL (Percona XtraDB Cluster).......State transfer in progress, setting sleep higher
.......... SUCCESS!
[root@pxc2 ~]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.6.20-68.0-56 Percona XtraDB Cluster (GPL), Release rel68.0, Revision 888, WSREP version 25.7, wsrep_25.7.r4126

Copyright (c) 2009-2014 Percona LLC and/or its affiliates
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from employees.employees limit 100;
ERROR 1146 (42S02): Table 'employees.employees' doesn't exist
mysql> use employees;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Error Log:
2014-10-19 04:25:33 6781 [Warning] InnoDB: Cannot open table employees/employees from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2014-10-19 04:25:44 6781 [Warning] InnoDB: Cannot open table employees/departments from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2014-10-19 04:25:44 6781 [Warning] InnoDB: Cannot open table employees/dept_emp from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2014-10-19 04:25:44 6781 [Warning] InnoDB: Cannot open table employees/dept_manager from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.
2014-10-19 04:25:44 6781 [Warning] InnoDB: Cannot open table employees/employees from the internal data dictionary of InnoDB though the .frm file for the table exists. See http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html for how you can resolve the problem.

[Raghavendra D Prabhu (raghavendra-prabhu)]

rsync SST doesn't support any special directories.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1018