| 2014-12-27 18:08:35 |
Raghavendra D Prabhu |
bug |
|
|
added bug |
| 2014-12-27 20:00:34 |
Raghavendra D Prabhu |
summary |
Fail early on donor on NON-PRIM |
Kill the SST donor/joiner thread in case of non-PRIM |
|
| 2014-12-27 20:01:54 |
Raghavendra D Prabhu |
description |
Possibly in a separate heartbeat thread, if the node goes into non-PRIM, fail.
Otherwise, this happens:
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp/tmp.q7sLLQ6Uy4 --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp/tmp.q7sLLQ6Uy4 --extra-lsndir='/tmp/tmp.q7sLLQ6Uy4' --stream=xbstream
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Waiting for ibbackup (pid=1332) to suspend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2'
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: uses posix_fadvise().
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: cd to /pxc/datadir
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: open files limit requested 0, set to 1048576
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using the following InnoDB configuration:
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_group_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_files_in_group = 2
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_file_size = 1073741824
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using O_DIRECT
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:17 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Generating a list of tablespaces
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock2-innobackupex-backup[18175]: Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./ibdata1
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_index_stats.ibd
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_worker_info.ibd
Dec 26 17:32:19 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:20 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:21 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:22 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:23 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:24 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:25 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:26 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_table_stats.ibd
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_relay_log_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_master_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./test/sbtest1.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Creating suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2' with pid '1332'
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Continuing after ibbackup has suspended
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Executing LOCK TABLES FOR BACKUP...
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 3038.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_query('HASH(0x2b41770)', 'LOCK TABLES FOR BACKUP') called at /usr/bin/innobackupex line 3440
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_lock_tables('HASH(0x2b41770)') called at /usr/bin/innobackupex line 1982
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::backup() called at /usr/bin/innobackupex line 1592
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Error:
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Error executing 'LOCK TABLES FOR BACKUP': DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Waiting for ibbackup (pid=1332) to finish
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18436]: innobackupex finished with error: 9. Check /pxc/datadir//innobackup.backup.log
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18437]: Cleanup after exit with status:22
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18439]: Cleaning up temporary directories
Full logs:
https://gist.github.com/defe6d152dbe88c34db5
https://gist.github.com/ffed1106e21592816d40 |
If the node goes into non-PRIM, fail.
Otherwise, this happens:
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp/tmp.q7sLLQ6Uy4 --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp/tmp.q7sLLQ6Uy4 --extra-lsndir='/tmp/tmp.q7sLLQ6Uy4' --stream=xbstream
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Waiting for ibbackup (pid=1332) to suspend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2'
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: uses posix_fadvise().
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: cd to /pxc/datadir
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: open files limit requested 0, set to 1048576
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using the following InnoDB configuration:
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_group_home_dir = ./
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_files_in_group = 2
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: innodb_log_file_size = 1073741824
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: using O_DIRECT
Dec 26 17:32:16 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:17 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Generating a list of tablespaces
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock2-innobackupex-backup[18175]: Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./ibdata1
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_index_stats.ibd
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:18 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_worker_info.ibd
Dec 26 17:32:19 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:20 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:21 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:22 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:23 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:24 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:25 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:26 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/innodb_table_stats.ibd
Dec 26 17:32:27 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_relay_log_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./mysql/slave_master_info.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] Streaming ./test/sbtest1.ibd
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: [01] ...done
Dec 26 17:32:28 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: >> log scanned up to (1636257)
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: xtrabackup: Creating suspend file '/tmp/tmp.q7sLLQ6Uy4/xtrabackup_suspended_2' with pid '1332'
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Continuing after ibbackup has suspended
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Executing LOCK TABLES FOR BACKUP...
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 3038.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_query('HASH(0x2b41770)', 'LOCK TABLES FOR BACKUP') called at /usr/bin/innobackupex line 3440
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::mysql_lock_tables('HASH(0x2b41770)') called at /usr/bin/innobackupex line 1982
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: main::backup() called at /usr/bin/innobackupex line 1592
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: innobackupex: Error:
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: Error executing 'LOCK TABLES FOR BACKUP': DBD::mysql::db do failed: Unknown command at /usr/bin/innobackupex line 3035.
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-innobackupex-backup[18155]: 141226 17:32:29 innobackupex: Waiting for ibbackup (pid=1332) to finish
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18436]: innobackupex finished with error: 9. Check /pxc/datadir//innobackup.backup.log
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18437]: Cleanup after exit with status:22
Dec 26 17:32:29 qaserver-04.ci.percona.com Dock1-wsrep-sst-donor[18439]: Cleaning up temporary directories
Full logs:
https://gist.github.com/defe6d152dbe88c34db5
https://gist.github.com/ffed1106e21592816d40 |
|
| 2015-01-06 12:15:33 |
Raghavendra D Prabhu |
nominated for series |
|
percona-xtradb-cluster/5.5 |
|
| 2015-01-06 12:15:33 |
Raghavendra D Prabhu |
bug task added |
|
percona-xtradb-cluster/5.5 |
|
| 2015-01-06 12:15:33 |
Raghavendra D Prabhu |
nominated for series |
|
percona-xtradb-cluster/5.6 |
|
| 2015-01-06 12:15:33 |
Raghavendra D Prabhu |
bug task added |
|
percona-xtradb-cluster/5.6 |
|
| 2015-01-06 12:16:11 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.6: milestone |
|
future-5.6 |
|
| 2015-01-06 12:16:12 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.5: milestone |
|
future-5.5 |
|
| 2015-01-06 12:16:16 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.6: assignee |
|
Raghavendra D Prabhu (raghavendra-prabhu) |
|
| 2015-01-06 12:16:17 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.5: assignee |
|
Raghavendra D Prabhu (raghavendra-prabhu) |
|
| 2015-01-06 12:16:21 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.6: status |
New |
Triaged |
|
| 2015-01-06 12:16:23 |
Raghavendra D Prabhu |
percona-xtradb-cluster/5.5: status |
New |
Triaged |
|
| 2015-11-18 10:05:18 |
Krunal Bauskar |
percona-xtradb-cluster/5.5: status |
Triaged |
Incomplete |
|
| 2015-11-18 10:05:21 |
Krunal Bauskar |
percona-xtradb-cluster/5.6: status |
Triaged |
Incomplete |
|
| 2015-11-18 10:05:23 |
Krunal Bauskar |
percona-xtradb-cluster/5.5: importance |
Undecided |
Medium |
|
| 2015-11-18 10:05:24 |
Krunal Bauskar |
percona-xtradb-cluster/5.6: importance |
Undecided |
Medium |
|
