The situation where rsync SST fails on both nodes with settings: datadir=/var/lib/mysql innodb_data_home_dir = /srv/mysql happens on both PXC 5.6.15 and 5.6.19 (testing repo). Moreover, when the directory specified in innodb_data_home_dir is missing on joiner node, not only the joiner node will fail to join the cluster, but also the donor node will get stuck in non-primary state (donor is the only working node). Err log snippet on joiner: "2014-06-17 14:43:49 15154 [Note] WSREP: Member 0.0 (percona20) requested state transfer from '*any*'. Selected 1.0 (percona10)(SYNCED) as donor. 2014-06-17 14:43:49 15154 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 1) 2014-06-17 14:43:49 15154 [Note] WSREP: Requesting state transfer: success, donor: 1 2014-06-17 14:43:49 15154 [Note] WSREP: (0589678f, 'tcp://0.0.0.0:4567') turning message relay requesting off 2014-06-17 14:44:00 15154 [Note] WSREP: 1.0 (percona10): State transfer to 0.0 (percona20) complete. 2014-06-17 14:44:00 15154 [Note] WSREP: Member 1.0 (percona10) synced with group. WSREP_SST: [INFO] Extracting binlog files: (20140617 14:44:00.801) percona10-bin.000014 ls: cannot access percona20-bin.*: No such file or directory WSREP_SST: [INFO] Joiner cleanup. (20140617 14:44:00.822) WSREP_SST: [INFO] Joiner cleanup done. (20140617 14:44:01.331) 2014-06-17 14:44:01 15154 [Note] WSREP: SST complete, seqno: 1 2014-06-17 14:44:01 15154 [Note] Plugin 'FEDERATED' is disabled. 2014-06-17 14:44:01 7fa1f935d7e0 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html. 2014-06-17 14:44:01 15154 [Note] InnoDB: Using atomics to ref count buffer pool pages 2014-06-17 14:44:01 15154 [Note] InnoDB: The InnoDB memory heap is disabled 2014-06-17 14:44:01 15154 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2014-06-17 14:44:01 15154 [Note] InnoDB: Compressed tables use zlib 1.2.3 2014-06-17 14:44:01 15154 [Note] InnoDB: Using Linux native AIO 2014-06-17 14:44:01 15154 [Note] InnoDB: Not using CPU crc32 instructions 2014-06-17 14:44:01 15154 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2014-06-17 14:44:01 15154 [Note] InnoDB: Completed initialization of buffer pool 2014-06-17 14:44:01 7fa1f935d7e0 InnoDB: Operating system error number 2 in a file operation. InnoDB: The error means the system cannot find the path specified. InnoDB: If you are installing InnoDB, remember that you must create InnoDB: directories yourself, InnoDB does not create them. 2014-06-17 14:44:01 15154 [ERROR] InnoDB: File /srv/mysql/ibdata1: 'create' returned OS error 71. Cannot continue operation 140617 14:44:01 mysqld_safe mysqld from pid file /var/lib/mysql/percona20.pid ended" Err log snippet on donor: "2014-06-17 14:43:50 3498 [Note] WSREP: Member 0.0 (percona20) requested state transfer from '*any*'. Selected 1.0 (percona10)(SYNCED) as donor. 2014-06-17 14:43:50 3498 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 1) 2014-06-17 14:43:50 3498 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-06-17 14:43:50 3498 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '192.168.3.30:4444/rsync_sst' --auth 'root:' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --binlog 'percona10-bin' --gtid '33fbfa96-f619-11e3-abd0-faf333aa6b5f:1'' 2014-06-17 14:43:50 3498 [Note] WSREP: sst_donor_thread signaled with 0 2014-06-17 14:43:50 3498 [Note] WSREP: Flushing tables for SST... 2014-06-17 14:43:50 3498 [Note] WSREP: Provider paused at 33fbfa96-f619-11e3-abd0-faf333aa6b5f:1 (12) 2014-06-17 14:43:50 3498 [Note] WSREP: Tables flushed. WSREP_SST: [INFO] Preparing binlog files for transfer: (20140617 14:43:50.471) percona10-bin.000014 2014-06-17 14:44:00 3498 [Note] WSREP: resuming provider at 12 2014-06-17 14:44:00 3498 [Note] WSREP: Provider resumed. 2014-06-17 14:44:01 3498 [Note] WSREP: 1.0 (percona10): State transfer to 0.0 (percona20) complete. 2014-06-17 14:44:01 3498 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1) 2014-06-17 14:44:01 3498 [Note] WSREP: Member 1.0 (percona10) synced with group. 2014-06-17 14:44:01 3498 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1) 2014-06-17 14:44:01 3498 [Note] WSREP: Synchronized with group, ready for connections 2014-06-17 14:44:01 3498 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-06-17 14:44:02 3498 [Note] WSREP: (3f134caf, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.3.30:4567 2014-06-17 14:44:03 3498 [Note] WSREP: (3f134caf, 'tcp://0.0.0.0:4567') reconnecting to 0589678f (tcp://192.168.3.30:4567), attempt 0 2014-06-17 14:44:07 3498 [Note] WSREP: evs::proto(3f134caf, OPERATIONAL, view_id(REG,0589678f,12)) suspecting node: 0589678f 2014-06-17 14:44:07 3498 [Note] WSREP: evs::proto(3f134caf, OPERATIONAL, view_id(REG,0589678f,12)) suspected node without join message, declaring inactive 2014-06-17 14:44:08 3498 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-06-17 14:44:08 3498 [Note] WSREP: Flow-control interval: [16, 16] 2014-06-17 14:44:08 3498 [Note] WSREP: Received NON-PRIMARY. 2014-06-17 14:44:08 3498 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1) 2014-06-17 14:44:08 3498 [Note] WSREP: New cluster view: global state: 33fbfa96-f619-11e3-abd0-faf333aa6b5f:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2 2014-06-17 14:44:08 3498 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-06-17 14:44:08 3498 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-06-17 14:44:08 3498 [Note] WSREP: Flow-control interval: [16, 16] 2014-06-17 14:44:08 3498 [Note] WSREP: Received NON-PRIMARY. 2014-06-17 14:44:08 3498 [Note] WSREP: New cluster view: global state: 33fbfa96-f619-11e3-abd0-faf333aa6b5f:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2 2014-06-17 14:44:08 3498 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-06-17 14:44:47 3498 [Note] WSREP: (3f134caf, 'tcp://0.0.0.0:4567') reconnecting to 0589678f (tcp://192.168.3.30:4567), attempt 30 2014-06-17 14:45:31 3498 [Note] WSREP: (3f134caf, 'tcp://0.0.0.0:4567') reconnecting to 0589678f (tcp://192.168.3.30:4567), attempt 60 2014-06-17 14:46:16 3498 [Note] WSREP: (3f134caf, 'tcp://0.0.0.0:4567') reconnecting to 0589678f (tcp://192.168.3.30:4567), attempt 90" Primary node is now trying to reconnect in an endless loop. Status: percona10 mysql> show status like 'ws%'; +------------------------------+--------------------------------------------------------+ | Variable_name | Value | +------------------------------+--------------------------------------------------------+ | wsrep_local_state_uuid | 33fbfa96-f619-11e3-abd0-faf333aa6b5f | | wsrep_protocol_version | 5 | | wsrep_last_committed | 1 | | wsrep_replicated | 0 | | wsrep_replicated_bytes | 0 | | wsrep_repl_keys | 0 | | wsrep_repl_keys_bytes | 0 | | wsrep_repl_data_bytes | 0 | | wsrep_repl_other_bytes | 0 | | wsrep_received | 14 | | wsrep_received_bytes | 1314 | | wsrep_local_commits | 0 | | wsrep_local_cert_failures | 0 | | wsrep_local_replays | 0 | | wsrep_local_send_queue | 0 | | wsrep_local_send_queue_avg | 0.000000 | | wsrep_local_recv_queue | 0 | | wsrep_local_recv_queue_avg | 0.071429 | | wsrep_local_cached_downto | 18446744073709551615 | | wsrep_flow_control_paused_ns | 0 | | wsrep_flow_control_paused | 0.000000 | | wsrep_flow_control_sent | 0 | | wsrep_flow_control_recv | 0 | | wsrep_cert_deps_distance | 0.000000 | | wsrep_apply_oooe | 0.000000 | | wsrep_apply_oool | 0.000000 | | wsrep_apply_window | 0.000000 | | wsrep_commit_oooe | 0.000000 | | wsrep_commit_oool | 0.000000 | | wsrep_commit_window | 0.000000 | | wsrep_local_state | 0 | | wsrep_local_state_comment | Initialized | | wsrep_cert_index_size | 0 | | wsrep_causal_reads | 0 | | wsrep_cert_interval | 0.000000 | | wsrep_incoming_addresses | 192.168.3.20:3306 | | wsrep_msg_repl_latency | avg=8.158e-06, sd=0, min=8.158e-06, max=8.158e-06, n=1 | | wsrep_cluster_conf_id | 18446744073709551615 | | wsrep_cluster_size | 1 | | wsrep_cluster_state_uuid | 33fbfa96-f619-11e3-abd0-faf333aa6b5f | | wsrep_cluster_status | non-Primary | | wsrep_connected | ON | | wsrep_local_bf_aborts | 0 | | wsrep_local_index | 0 | | wsrep_provider_name | Galera | | wsrep_provider_vendor | Codership Oy