SST encryption no working with xtrabackup-v2

[cortocopy]

On a fresh install of Percona Galera (CentOS 7, Percona-XtraDB-Cluster-56.x86_64) I'm unable to join a node with xtrabackup-v2 and openssl encryption.

I'm using the most basic my.cnf as per the Galera main documentation (5.6.24-25.11) to narrow down possible causes.

I'm also using the test files from Launchpad as linked in the docs for testing purposes. Permissions for these files are 0600 and owned by user mysql

The log says that there is a problem with the socat connection as “library:fopen:Permission denied”. However, I'm not sure what the denial is for.

I've elimiated usual suspects by:
- Disabling selinux
- Opening ports 3306/tcp, 4444/tcp, 4567/tcp, 4568/tcp in firewalld
- Ensuring that socat is built with openSSL:

socat -V | grep OPENSSL
#define WITH_OPENSSL 1

- Even giving a 777 to the .pem and .crt files

These are the my.conf file and log

##################MY.CONF

[mysqld]
datadir=/var/lib/mysql
user=mysql
log_error = /var/log/mysql/error.log

# Path to Galera library
wsrep_provider=/usr/lib64/libgalera_smm.so

# Cluster connection URL contains the IPs of node#1, node#2 and node#3
wsrep_cluster_address=gcomm://node#1.IP, node#2.IP, node#3.IP

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=node.domain.com

# SST method
wsrep_sst_method=xtrabackup-v2

# Cluster name
wsrep_cluster_name=my_centos_cluster

[xtrabackup]
target_dir = /data/backups/mysql/

[sst]
encrypt=2
tca=/etc/ssh/mysql/node1.crt
tcert=/etc/ssh/mysql/node1.pem

!includedir /etc/my.cnf.d

##################LOG (extract)

2015-07-06 10:51:20 10739 [Note] WSREP: Prepared SST request: xtrabackup-v2|node#2.IP:4444/xtrabackup_sst//1
2015-07-06 10:51:20 10739 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-07-06 10:51:20 10739 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-07-06 10:51:20 10739 [Note] WSREP: Service thread queue flushed.
2015-07-06 10:51:20 10739 [Note] WSREP: Assign initial position for certification: 15, protocol version: 3
2015-07-06 10:51:20 10739 [Note] WSREP: Service thread queue flushed.
2015-07-06 10:51:20 10739 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (0d8af775-23ea-11e5-9280-3bd6$
         at galera/src/replicator_str.cpp:prepare_for_IST():463. IST will be unavailable.
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u openssl-listen:4444,reuseaddr,cert=/etc/ssh/mysql/node1.pem,cafile=/etc/ssh/mysql/node1.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150706 10:51:$
2015/07/06 10:51:20 socat[10991] E SSL_CTX_load_verify_locations(): error:0200100D:system library:fopen:Permission denied
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20150706 10:51:20.535)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150706 10:51:20.536)

[Sveta Smirnova (svetasmirnova)]

Thank you for the report.

Error message says: "WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u openssl-listen:4444,reuseaddr,cert=/etc/ssh/mysql/node1.pem,cafile=/etc/ssh/mysql/node1.crt stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150706 10:51:$
2015/07/06 10:51:20 socat[10991] E SSL_CTX_load_verify_locations(): error:0200100D:system library:fopen:Permission denied" What are permissions of /etc/ssh/mysql/node1.pem and /etc/ssh/mysql/node1.crt files? Which user you run mysqld as?

[cortocopy (cortocopy)]

I don't recall the exact details but I managed to sort it out, but I managed in the end with different blocks (sst is also now in my config so that may have been it)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1835