WSREP: 3.0: State transfer to 1.0 failed: -1 (Operation not permitted)

[Thomas Roog]

Hi,

I can't initialize new node, I am getting the same error all the time:
State transfer to 1.0 (10.1.160.11) failed: -1 (Operation not permitted)
Thank you in advance for any help.

root@primysql11:/home/tomaszro# service mysql start

--------------------------------------------------------------------------------------------------------------------------

* Stale sst_in_progress file in datadir mysqld
* Starting MySQL (Percona XtraDB Cluster) database server mysqld
* State transfer in progress, setting sleep higher mysqld
* The server quit without updating PID file (/data01/mysql/mysqld.pid).

donor

----------------------------------------------------------------------------------------------------------

2015-06-30 16:56:59 7281 [Note] WSREP: New cluster view: global state: 146797dd-71a9-11e4-98e5-3ebe51e67dbd:37116929, view# 205: Primary, number of nodes: 4, my index: 3, protocol version 3
2015-06-30 16:57:00 7281 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-06-30 16:57:00 7281 [Note] WSREP: Service thread queue flushed.
2015-06-30 16:57:00 7281 [Note] WSREP: Assign initial position for certification: 37116929, protocol version: 3
2015-06-30 16:57:00 7281 [Note] WSREP: Service thread queue flushed.
2015-06-30 16:57:00 7281 [Note] WSREP: Member 1.0 (10.1.160.11) requested state transfer from '10.1.160.7,10.1.160.6'. Selected 3.0 (10.1.160.7)(SYNCED) as donor.
2015-06-30 16:57:00 7281 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 37116932)
2015-06-30 16:57:00 7281 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.1.160.11:4444/xtrabackup_sst//1' --auth 'root:password' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data01/mysql/' --defaults-file '/etc/mysql/my.cnf' '' --gtid '146797dd-71a9-11e4-98e5-3ebe51e67dbd:37116932''
2015-06-30 16:57:00 7281 [Note] WSREP: sst_donor_thread signaled with 0
2015-06-30 16:57:00 7281 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.1.160.11:4444/xtrabackup_sst//1' --auth 'root:password' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data01/mysql/' --defaults-file '/etc/mysql/my.cnf' '' --gtid '146797dd-71a9-11e4-98e5-3ebe51e67dbd:37116932'
2015-06-30 16:57:00 7281 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.1.160.11:4444/xtrabackup_sst//1' --auth 'root:password' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data01/mysql/' --defaults-file '/etc/mysql/my.cnf' '' --gtid '146797dd-71a9-11e4-98e5-3ebe51e67dbd:37116932': 1 (Operation not permitted)
2015-06-30 16:57:00 7281 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.1.160.11:4444/xtrabackup_sst//1' --auth 'root:password' --socket '/var/run/mysqld/mysqld.sock' --datadir '/data01/mysql/' --defaults-file '/etc/mysql/my.cnf' '' --gtid '146797dd-71a9-11e4-98e5-3ebe51e67dbd:37116932'
2015-06-30 16:57:00 7281 [Warning] WSREP: 3.0 (10.1.160.7): State transfer to 1.0 (10.1.160.11) failed: -1 (Operation not permitted)
2015-06-30 16:57:00 7281 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 37116932)
2015-06-30 16:57:00 7281 [Note] WSREP: declaring 23f6172f at tcp://10.1.160.6:4567 stable
2015-06-30 16:57:00 7281 [Note] WSREP: declaring 43cf320e at tcp://10.1.160.10:4567 stable
2015-06-30 16:57:00 7281 [Note] WSREP: forgetting 419ce10d (tcp://10.1.160.11:4567)

joiner

-------------------------------------------------------------------------------------------------------------------------

2015-06-30 17:18:01 23319 [Note] WSREP: Flow-control interval: [32, 32]
2015-06-30 17:18:01 23319 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 37120605)
2015-06-30 17:18:01 23319 [Note] WSREP: State transfer required:
Group state: 146797dd-71a9-11e4-98e5-3ebe51e67dbd:37120605
Local state: 00000000-0000-0000-0000-000000000000:-1
2015-06-30 17:18:01 23319 [Note] WSREP: New cluster view: global state: 146797dd-71a9-11e4-98e5-3ebe51e67dbd:37120605, view# 207: Primary, number of nodes: 4, my index: 1, protocol version 3
2015-06-30 17:18:01 23319 [Warning] WSREP: Gap in state sequence. Need state transfer.
2015-06-30 17:18:01 23319 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.1.160.11' --auth 'root:password' --datadir '/data01/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '23319' '' '
WSREP_SST: [INFO] Streaming with xbstream (20150630 17:18:01.976)
WSREP_SST: [INFO] Using socat as streamer (20150630 17:18:01.980)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150630 17:18:02.040)
2015-06-30 17:18:02 23319 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.1.160.11:4444/xtrabackup_sst//1
2015-06-30 17:18:02 23319 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-06-30 17:18:02 23319 [Note] WSREP: Service thread queue flushed.
2015-06-30 17:18:02 23319 [Note] WSREP: Assign initial position for certification: 37120605, protocol version: 3
2015-06-30 17:18:02 23319 [Note] WSREP: Service thread queue flushed.
2015-06-30 17:18:02 23319 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (146797dd-71a9-11e4-98e5-3ebe51e67dbd): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.
2015-06-30 17:18:02 23319 [Note] WSREP: Member 1.0 (10.1.160.11) requested state transfer from '10.1.160.10,10.1.160.7,10.1.160.6'. Selected 2.0 (10.1.160.10)(SYNCED) as donor.
2015-06-30 17:18:02 23319 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 37120606)
2015-06-30 17:18:02 23319 [Note] WSREP: Requesting state transfer: success, donor: 2
2015-06-30 17:18:02 23319 [Warning] WSREP: 2.0 (10.1.160.10): State transfer to 1.0 (10.1.160.11) failed: -1 (Operation not permitted)
2015-06-30 17:18:02 23319 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():731: Will never receive state. Need to abort.
2015-06-30 17:18:02 23319 [Note] WSREP: gcomm: terminating thread
2015-06-30 17:18:02 23319 [Note] WSREP: gcomm: joining thread
2015-06-30 17:18:02 23319 [Note] WSREP: gcomm: closing backend
2015-06-30 17:18:02 23319 [Note] WSREP: view(view_id(NON_PRIM,23f6172f,217) memb {
3215e5ba,0
} joined {
} left {
} partitioned {
23f6172f,0
43cf320e,0
81113083,0
})
2015-06-30 17:18:02 23319 [Note] WSREP: view((empty))
2015-06-30 17:18:02 23319 [Note] WSREP: gcomm: closed
2015-06-30 17:18:02 23319 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Aborted (core dumped)

oiner config file

------------------------------------------------------------

# TR 2015-01-20 Specified order of donors
# TR 2015-01-19 Updated and merged with Percona recommendation and own experience
# All files in this package is subject to the GPL v2 license
# More information is in the COPYING file in the top directory of this package.
# Copyright (C) 2011 severalnines.com

[MYSQLD]
user=mysql
basedir=/usr/
datadir=/data01/mysql
socket=/var/run/mysqld/mysqld.sock
pid_file=mysqld.pid
port=3306
log_error=/var/log/mysql.log
log_warnings=2
# Use only ipv4 for connections
bind_address=0.0.0.0
#log_output=FILE
### INNODB OPTIONS
innodb_buffer_pool_size=12G
innodb_flush_log_at_trx_commit=2
innodb_file_per_table=1
innodb_data_file_path = ibdata1:100M:autoextend
## You may want to tune the below depending on number of cores and disk sub
#innodb_read_io_threads=4
#innodb_write_io_threads=4
innodb_io_capacity=200
innodb_doublewrite=1
innodb_log_file_size=2G
innodb_log_buffer_size=32M
innodb_buffer_pool_instances=4
innodb_log_files_in_group=2
innodb_thread_concurrency=0
innodb_open_files=2000
#innodb_file_format=barracuda
innodb_flush_method=O_DIRECT
innodb_autoinc_lock_mode=2
## avoid statistics update when doing e.g show tables
innodb_stats_on_metadata=0
default_storage_engine=innodb

## HEAVY WRITING ENVIROMENT ##
innodb_max_dirty_pages_pct = 90

## SPEEDUP ##
## /dev/shm uses virtual memory instead of a persistent storage device
tmpdir=/tmp
back_log=2048

## EVOLVE SPECIC SETTINGS ##
lower_case_table_names = 1
#autocommit=0
init_connect='SET autocommit=0'

## DEFAULT CHARACTER SET ##
skip-character-set-client-handshake
collation_server=utf8_general_ci
character_set_server=utf8

# REPLICATION SPECIFIC _ GENERAL
#server_id must be unique across all mysql servers participating in replication.
server_id=110
# REPLICATION SPECIFIC
binlog_format=ROW
log_bin=binlog
binlog_cache_size=1M
#relay_log=relay-bin
slave_net_timeout=60
#skip-slave-start=1
#read_only=1
expire_logs_days=7
sync_binlog=0
log_slave_updates=1
#gtid_mode=ON
# allow create users fuctions
log_bin_trust_function_creators=1
#enforce_gtid_consistency=1
# OTHER THINGS, BUFFERS ETC
key_buffer_size = 32M
tmp_table_size = 64M
max_heap_table_size = 64M
## SAFETY ##
max_allowed_packet = 16M
max_connect_errors = 1000000
#sort_buffer_size = 256K
#read_buffer_size = 256K
#read_rnd_buffer_size = 512K
#myisam_sort_buffer_size = 8M
skip_name_resolve
memlock=0
sysdate_is_now=1
max_connections=200
## EXTRA SAFETY ##
symbolic-links=0
thread_cache_size=512
open_files_limit=65535
table_definition_cache=4096
query_cache_type = 0
query_cache_size = 0
table_open_cache=10000
# 5.6 backwards compatibility
explicit_defaults_for_timestamp=1

## SLOW LOG ##
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
log_slow_verbosity = FULL
long_query_time = 10
slow_query_log_use_global_control = long_query_time,log_slow_filter,log_slow_rate_limit,log_slow_verbosity
log_queries_not_using_indexes=1
event_scheduler=0

##
## WSREP options
##

# Full path to wsrep provider library or 'none'
wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_node_address=10.1.160.11
# Provider specific configuration options
wsrep_provider_options="gcache.size=4096M; gmcast.segment=0"

# Logical cluster name. Should be the same for all nodes.
wsrep_cluster_name="my_wsrep_cluster"

# Group communication system handle
wsrep_cluster_address=gcomm://10.1.160.6:4567,10.1.160.7:4567,10.1.160.10:4567,10.1.160.11:4567

# Human_readable node name (non-unique). Hostname by default.
wsrep_node_name=10.1.160.11

# Address for incoming client connections. Autodetect by default.
#wsrep_node_incoming_address=

# How many threads will process writesets from other nodes
#wsrep_slave_threads=4

# DBUG options for wsrep provider
#wsrep_dbug_option

# Generate fake primary keys for non-PK tables (required for multi-master
# and parallel applying operation)
wsrep_certify_nonPK=1

# Location of the directory with data files. Needed for non-mysqldump
# state snapshot transfers. Defaults to mysql_real_data_home.
#wsrep_data_home_dir=

# Maximum number of rows in write set
wsrep_max_ws_rows=131072

# Maximum size of write set
wsrep_max_ws_size=1073741824

# to enable debug level logging, set this to 1
wsrep_debug=0

# convert locking sessions into transactions
wsrep_convert_LOCK_to_trx=0

# how many times to retry deadlocked autocommits
wsrep_retry_autocommit=1

# change auto_increment_increment and auto_increment_offset automatically
wsrep_auto_increment_control=1

# replicate myisam
wsrep_replicate_myisam=1
# retry autoinc insert, which failed for duplicate key error
wsrep_drupal_282555_workaround=0

# enable "strictly synchronous" semantics for read operations
wsrep_causal_reads=0

# Command to call when node status or cluster membership changes.
# Will be passed all or some of the following options:
# --status - new status of this node
# --uuid - UUID of the cluster
# --primary - whether the component is primary or not ("yes"/"no")
# --members - comma-separated list of members
# --index - index of this node in the list
wsrep_notify_cmd=/usr/bin/wsrep_notify_cc

##
## WSREP State Transfer options
##

# State Snapshot Transfer method
# ClusterControl currently DOES NOT support wsrep_sst_method=mysqldump
wsrep_sst_method=xtrabackup-v2

# Address on THIS node to receive SST at. DON'T SET IT TO DONOR ADDRESS!!!
# (SST method dependent. Defaults to the first IP of the first interface)
#wsrep_sst_receive_address=

# SST authentication string. This will be used to send SST to joining nodes.
# Depends on SST method. For mysqldump method it is root:<root password>
wsrep_sst_auth=root:password

# Desired SST donor name.
wsrep_sst_donor=10.1.160.10,10.1.160.7,10.1.160.6

# Protocol version to use
# wsrep_protocol_version=

[MYSQL]
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[client]
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[mysqldump]
max_allowed_packet = 16M
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[MYSQLD_SAFE]
pid_file=mysqld.pid
log_error=/var/log/mysql.log
basedir=/usr/
datadir=/data01/mysql

donor config file

------------------------------------------------------------------------------

# TR 2014-01-20 Specified order of donors
# TR 2014-01-19
# All files in this package is subject to the GPL v2 license
# More information is in the COPYING file in the top directory of this package.
# Copyright (C) 2011 severalnines.com

[MYSQLD]
user=mysql
basedir=/usr/
datadir=/data01/mysql
socket=/var/run/mysqld/mysqld.sock
pid_file=mysqld.pid
port=3306
log_error=/var/log/mysql.log
log_warnings=2
# Use only ipv4 for connections
bind_address=0.0.0.0
#log_output=FILE
### INNODB OPTIONS
innodb_buffer_pool_size=12G
innodb_flush_log_at_trx_commit=2
innodb_file_per_table=1
innodb_data_file_path = ibdata1:100M:autoextend
## You may want to tune the below depending on number of cores and disk sub
#innodb_read_io_threads=4
#innodb_write_io_threads=4
innodb_io_capacity=200
innodb_doublewrite=1
innodb_log_file_size=2G
innodb_log_buffer_size=32M
innodb_buffer_pool_instances=4
innodb_log_files_in_group=2
innodb_thread_concurrency=0
innodb_open_files=2000
#innodb_file_format=barracuda
innodb_flush_method=O_DIRECT
innodb_autoinc_lock_mode=2
## avoid statistics update when doing e.g show tables
innodb_stats_on_metadata=0
default_storage_engine=innodb

## HEAVY WRITING ENVIROMENT ##
innodb_max_dirty_pages_pct = 90

## SPEEDUP ##
## /dev/shm uses virtual memory instead of a persistent storage device
tmpdir=/tmp
back_log=2048

## EVOLVE SPECIC SETTINGS ##
lower_case_table_names = 1
#autocommit=0
init_connect='SET autocommit=0'

## DEFAULT CHARACTER SET ##
skip-character-set-client-handshake
collation_server=utf8_general_ci
character_set_server=utf8

# REPLICATION SPECIFIC _ GENERAL
#server_id must be unique across all mysql servers participating in replication.
server_id=7
# REPLICATION SPECIFIC
binlog_format=ROW
log_bin=binlog
binlog_cache_size=1M
#relay_log=relay-bin
slave_net_timeout=60
#skip-slave-start=1
#read_only=1
expire_logs_days=7
sync_binlog=0
log_slave_updates=1
#gtid_mode=ON
# allow create users fuctions
log_bin_trust_function_creators=1
#enforce_gtid_consistency=1
# OTHER THINGS, BUFFERS ETC
key_buffer_size = 32M
tmp_table_size = 64M
max_heap_table_size = 64M
## SAFETY ##
max_allowed_packet = 16M
max_connect_errors = 1000000
#sort_buffer_size = 256K
#read_buffer_size = 256K
#read_rnd_buffer_size = 512K
#myisam_sort_buffer_size = 8M
skip_name_resolve
memlock=0
sysdate_is_now=1
max_connections=200
## EXTRA SAFETY ##
symbolic-links=0
thread_cache_size=512
open_files_limit=65535
table_definition_cache=4096
query_cache_type = 0
query_cache_size = 0
table_open_cache=10000
# 5.6 backwards compatibility
explicit_defaults_for_timestamp=1

## SLOW LOG ##
slow_query_log = 1
slow_query_log_file = /var/log/mysql/mysql-slow.log
log_slow_verbosity = FULL
long_query_time = 10
slow_query_log_use_global_control = long_query_time,log_slow_filter,log_slow_rate_limit,log_slow_verbosity
log_queries_not_using_indexes=1
event_scheduler=0

##
## WSREP options
##

# Full path to wsrep provider library or 'none'
wsrep_provider=/usr/lib/libgalera_smm.so

wsrep_node_address=10.1.160.7
# Provider specific configuration options
wsrep_provider_options="gcache.size=4096M; gmcast.segment=0"

# Logical cluster name. Should be the same for all nodes.
wsrep_cluster_name="my_wsrep_cluster"

# Group communication system handle
wsrep_cluster_address=gcomm://10.1.160.10:4567,10.1.160.6:4567,10.1.160.7:4567

# Human_readable node name (non-unique). Hostname by default.
wsrep_node_name=10.1.160.7

# Address for incoming client connections. Autodetect by default.
#wsrep_node_incoming_address=

# How many threads will process writesets from other nodes
#wsrep_slave_threads=4

# DBUG options for wsrep provider
#wsrep_dbug_option

# Generate fake primary keys for non-PK tables (required for multi-master
# and parallel applying operation)
wsrep_certify_nonPK=1

# Location of the directory with data files. Needed for non-mysqldump
# state snapshot transfers. Defaults to mysql_real_data_home.
#wsrep_data_home_dir=

# Maximum number of rows in write set
wsrep_max_ws_rows=131072

# Maximum size of write set
wsrep_max_ws_size=1073741824

# to enable debug level logging, set this to 1
wsrep_debug=0

# convert locking sessions into transactions
wsrep_convert_LOCK_to_trx=0

# how many times to retry deadlocked autocommits
wsrep_retry_autocommit=1

# change auto_increment_increment and auto_increment_offset automatically
wsrep_auto_increment_control=1

# replicate myisam
wsrep_replicate_myisam=1
# retry autoinc insert, which failed for duplicate key error
wsrep_drupal_282555_workaround=0

# enable "strictly synchronous" semantics for read operations
wsrep_causal_reads=0

# Command to call when node status or cluster membership changes.
# Will be passed all or some of the following options:
# --status - new status of this node
# --uuid - UUID of the cluster
# --primary - whether the component is primary or not ("yes"/"no")
# --members - comma-separated list of members
# --index - index of this node in the list
wsrep_notify_cmd=/usr/bin/wsrep_notify_cc

##
## WSREP State Transfer options
##

# State Snapshot Transfer method
# ClusterControl currently DOES NOT support wsrep_sst_method=mysqldump
wsrep_sst_method=xtrabackup-v2

# Address on THIS node to receive SST at. DON'T SET IT TO DONOR ADDRESS!!!
# (SST method dependent. Defaults to the first IP of the first interface)
#wsrep_sst_receive_address=

# SST authentication string. This will be used to send SST to joining nodes.
# Depends on SST method. For mysqldump method it is root:<root password>
wsrep_sst_auth=root:password

# Desired SST donor name.
wsrep_sst_donor=10.1.160.10,10.1.160.6

# Protocol version to use
# wsrep_protocol_version=

[MYSQL]
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[client]
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[mysqldump]
max_allowed_packet = 16M
socket=/var/run/mysqld/mysqld.sock
#default_character_set=utf8

[MYSQLD_SAFE]
pid_file=mysqld.pid
log_error=/var/log/mysql.log
basedir=/usr/
datadir=/data01/mysql

donor versions

----------------------------------------------------------------
percona-nagios-plugins 1.1.4-1
percona-toolkit 2.2.12
percona-xtrabackup 2.2.11-1.trusty
percona-xtradb-cluster-client-5.6 5.6.22-25.8-978.trusty
percona-xtradb-cluster-common-5.6 5.6.22-25.8-978.trusty
percona-xtradb-cluster-galera-3.x 3.9.3494.trusty
percona-xtradb-cluster-server-5.6 5.6.22-25.8-978.trusty

joiner versions (yes, they are a little above, but this worked on other env without any problems many times)

--------------------------------------------------------------------------------------

percona-xtrabackup 2.2.11-1.trusty
percona-xtradb-cluster-client-5.6 5.6.24-25.11-1.trusty
percona-xtradb-cluster-common-5.6 5.6.24-25.11-1.trusty
percona-xtradb-cluster-galera-3.x 3.11.1.trusty
percona-xtradb-cluster-server-5.6 5.6.24-25.11-1.trusty

donor

------------------------------------------------------------------------
ll /etc/mysql/*.cnf
-rw-r--r-- 1 mysql mysql 333 Nov 21 2014 /etc/mysql/debian.cnf
-rw------- 1 mysql mysql 39 Jun 30 17:17 /etc/mysql/my_cmon.cnf
-rw-r--r-- 1 mysql mysql 6048 Jun 30 17:17 /etc/mysql/my.cnf

ll /usr/bin/wsrep*
-rwxr-xr-x 1 mysql mysql 2163 Jun 30 17:17 /usr/bin/wsrep_notify_cc*
-rwxrwxrwx 1 root root 3496 Jun 30 15:33 /usr/bin/wsrep_sst_common*
-rwxrwxrwx 1 root root 5626 Jun 30 15:33 /usr/bin/wsrep_sst_mysqldump*
-rwxrwxrwx 1 root root 10051 Jun 30 15:33 /usr/bin/wsrep_sst_rsync*
-rwxrwxrwx 1 root root 21632 Jun 30 15:33 /usr/bin/wsrep_sst_xtrabackup*
-rwxrwxrwx 1 root root 30277 Jun 30 16:14 /usr/bin/wsrep_sst_xtrabackup-v2*

joiner

------------------------------------------------

ll /usr/bin/wsrep*
-rwxr-xr-x 1 mysql mysql 2164 Jun 30 17:17 /usr/bin/wsrep_notify_cc*
-rwxr-xr-x 1 root root 3496 Mar 9 10:34 /usr/bin/wsrep_sst_common*
-rwxr-xr-x 1 root root 5626 Mar 9 10:34 /usr/bin/wsrep_sst_mysqldump*
-rwxr-xr-x 1 root root 10051 Mar 9 10:34 /usr/bin/wsrep_sst_rsync*
lrwxrwxrwx 1 root root 26 May 6 17:10 /usr/bin/wsrep_sst_rsync_wan -> /usr/bin///wsrep_sst_rsync*
-rwxr-xr-x 1 root root 21632 Mar 9 10:34 /usr/bin/wsrep_sst_xtrabackup*
-rwxr-xr-x 1 root root 30223 Mar 9 10:34 /usr/bin/wsrep_sst_xtrabackup-v2*

ll /etc/mysql/*.cnf
-rw------- 1 mysql mysql 333 May 6 17:09 /etc/mysql/debian.cnf
-rw------- 1 mysql mysql 39 Jun 30 17:17 /etc/mysql/my_cmon.cnf
-rw-r--r-- 1 mysql mysql 6115 Jun 30 17:17 /etc/mysql/my.cnf

[David Bennett (dbpercona)]

Is the wsrep_sst_auth value correct? Is this the correct root password on the donor?

[Thomas Roog (thomas.roog)]

Hi David, thank you for asking. Yes, it is.

From joiner to the donor
-------------------------------------------------------------------
/home/tomaszro# mysql -h10.1.160.7 -uroot -ppassword -e "select now();"
Warning: Using a password on the command line interface can be insecure.
+---------------------+
| now() |
+---------------------+
| 2015-06-30 17:43:23 |
+---------------------+

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1834