Kill the SST donor/joiner thread in case of non-PRIM

[Raghavendra D Prabhu]

If the node goes into non-PRIM, fail.

Otherwise, this happens:

Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp/tmp.q7sLLQ6Uy4 --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp/tmp.q7sLLQ6Uy4 --extra-lsndir='/tmp/tmp.q7sLLQ6Uy4' --stream=xbstream
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Waiting for ibbackup (pid=1332) to suspend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2'
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: uses posix_fadvise().
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: cd to /pxc/datadir
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: open files limit requested 0, set to 1048576
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using the following InnoDB configuration:
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_group_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_files_in_group = 2
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_file_size = 1073741824
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using O_DIRECT
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:17 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Generating a list of tablespaces
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock2-innobackupex-backup[18175]: Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./ibdata1
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_index_stats.ibd
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_worker_info.ibd
Dec 26 17:32:19 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:20 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:21 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:22 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:23 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:24 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:25 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:26 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_table_stats.ibd
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_relay_log_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_master_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./test/sbtest1.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Creating suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2' with pid '1332'
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Continuing after ibbackup has suspended
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Executing LOCK TABLES FOR BACKUP...
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 3038.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_query('HASH(0x2b41770)', 'LOCK TABLES FOR BACKUP') called at /usr/bin/innobackupex line 3440
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_lock_tables('HASH(0x2b41770)') called at /usr/bin/innobackupex line 1982
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::backup() called at /usr/bin/innobackupex line 1592
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Error:
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Error executing 'LOCK TABLES FOR BACKUP': DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Waiting for ibbackup (pid=1332) to finish
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18436]: innobackupex finished with error: 9. Check /pxc/datadir//innobackup.backup.log
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18437]: Cleanup after exit with status:22
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18439]: Cleaning up temporary directories

Full logs:

https://gist.github.com/defe6d152dbe88c34db5
https://gist.github.com/ffed1106e21592816d40

[Krunal Bauskar (krunal-bauskar)]

Can we have a use-case and error logs ?

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1118