Soft failure/warning on SST dependencies

[jolan]

relevant line:

/usr//bin/wsrep_sst_xtrabackup-v2: line 95: pv: command not found

surrounding lines:

WSREP_SST: [INFO] Evaluating socat -u openssl-listen:4444,reuseaddr,cert=/etc/mysql/ssl/client.pem,cafile=/etc/mysql/ssl/server.crt stdio | pv -f -i 10 -N joiner | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20131106 13:43:12.010)
/usr//bin/wsrep_sst_xtrabackup-v2: line 95: pv: command not found
2013/11/06 13:43:12 socat[5278] E SSL_CTX_load_verify_locations(): error:0200100D:system library:fopen:Permission denied
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 127 0 (20131106 13:43:12.041)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20131106 13:43:12.047)
WSREP_SST: [INFO] Removing the sst_in_progress file (20131106 13:43:12.051)
131106 13:43:12 [Note] WSREP: Prepared SST request: xtrabackup-v2|23.92.31.36:4444/xtrabackup_sst
131106 13:43:12 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131106 13:43:12 [Note] WSREP: Assign initial position for certification: 1415, protocol version: 2
131106 13:43:12 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (cfe05aee-4717-11e3-a676-5a7b9fdcb458): 1 (Operation not permitted)
  at galera/src/replicator_str.cpp:prepare_for_IST():436. IST will be unavailable.
131106 13:43:12 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '23.92.31.36' --auth 'root:*************' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '5029': 32 (Broken pipe)
131106 13:43:12 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
131106 13:43:12 [ERROR] WSREP: SST failed: 32 (Broken pipe)
131106 13:43:12 [ERROR] Aborting

[jolan (jolan)]

I think this is due to:

[sst]
progress=1

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Jolan,

Yes, it is indeed due to that, the reason that it is not a dependency is that it is enabled only when certain options (like pv) are set.

More details on that (and other dependencies) here - http://www.percona.com/doc/percona-xtradb-cluster/manual/xtrabackup_sst.html

[Raghavendra D Prabhu (raghavendra-prabhu)]

Created https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1250326 for this.

[David Bennett (dbpercona)]

I just spent time (and not the first time) trying to figure out why certain SST jobs were failing and this was ultimately the cause.

The error report on this is easy to miss.

I agree that we should not make it an installation depenency. Especially since pv is not a mainstream package in some distributions.

However, it might be feasable to simply disable the progress feature if the pv command is not found in the path.

This could be accomplished in the wsrep_sst_xtrabackup* scripts by checking for pv using which and if not found, display and error message that progress reports will be disabled.

It would probably be not too difficult to place is as a insert_pv function in wsrep_sst_common that handled the insertion of the pipe viewer into the command before execution and simply displayed a disable warning if not found.

Since the progress feature is not a critial path to the proper operation of Cluster replication, I am thinking that the lack of the pv command should probably not prevent a cluster node from syncing.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@David,

Agreed. For additional utilities like pv which may not affect critical path of SST, we can make it warn than fail.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1510