SST xtrabackup in 5.5.31 is not clobbering JOINER datadir
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL patches by Codership |
New
|
Undecided
|
Unassigned | ||
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC |
Fix Released
|
High
|
Raghavendra D Prabhu |
Bug Description
3 node cluster with table 'test.sbtest1'.
All nodes report:
[root@ip-
-rw-rw----. 1 mysql mysql 8632 Aug 16 12:40 /var/lib/
-rw-rw----. 1 mysql mysql 214958080 Aug 16 12:41 /var/lib/
[root@ip-
-rw-rw----. 1 mysql mysql 8632 Aug 16 12:40 /var/lib/
-rw-rw----. 1 mysql mysql 243269632 Aug 16 12:41 /var/lib/
[root@ip-
-rw-rw----. 1 mysql mysql 8632 Aug 16 12:40 /var/lib/
-rw-rw----. 1 mysql mysql 243269632 Aug 16 12:41 /var/lib/
Now, shutdown last node:
[root@ip-
Shutting down MySQL (Percona XtraDB Cluster)..... SUCCESS!
Drop the table on one of the other nodes:
ip-10-32-231-30 mysql> drop table test.sbtest1;
Query OK, 0 rows affected (0.10 sec)
I can see table is gone on running nodes, still present on offline node (as expected):
[root@ip-
ls: cannot access /var/lib/
[root@ip-
ls: cannot access /var/lib/
[root@ip-
-rw-rw----. 1 mysql mysql 8632 Aug 16 12:40 /var/lib/
-rw-rw----. 1 mysql mysql 289406976 Aug 16 12:41 /var/lib/
So, I remove grastate.dat to force SST and restart last node:
[root@ip-
rm: remove regular file `/var/lib/
[root@ip-
Starting MySQL (Percona XtraDB Cluster)....SST in progress, setting sleep higher....
I can see innodbackupex running, so this is using xtrabackup and is does full SST. Node comes up and joins the cluster, and yet I still see the tablespace from sbtest1:
[root@ip-
ls: cannot access /var/lib/
[root@ip-
ls: cannot access /var/lib/
[root@ip-
-rw-rw----. 1 mysql mysql 8632 Aug 16 12:40 /var/lib/
-rw-rw----. 1 mysql mysql 289406976 Aug 16 12:41 /var/lib/
ip-10-40-197-24 mysql> show global status like 'wsrep%';
+------
| Variable_name | Value |
+------
| wsrep_local_
| wsrep_protocol_
| wsrep_last_
| wsrep_replicated | 0 |
| wsrep_replicate
| wsrep_received | 3 |
| wsrep_received_
| wsrep_local_commits | 0 |
| wsrep_local_
| wsrep_local_
| wsrep_local_replays | 0 |
| wsrep_local_
| wsrep_local_
| wsrep_local_
| wsrep_local_
| wsrep_flow_
| wsrep_flow_
| wsrep_flow_
| wsrep_cert_
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 0.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 0.000000 |
| wsrep_local_state | 4 |
| wsrep_local_
| wsrep_cert_
| wsrep_causal_reads | 0 |
| wsrep_incoming_
| wsrep_cluster_
| wsrep_cluster_size | 3 |
| wsrep_cluster_
| wsrep_cluster_
| wsrep_connected | ON |
| wsrep_local_index | 2 |
| wsrep_provider_name | Galera |
| wsrep_provider_
| wsrep_provider_
| wsrep_ready | ON |
+------
40 rows in set (0.00 sec)
My configuration is thus:
[mysqld]
datadir = /var/lib/mysql
log_error = error.log
key_buffer_size = 128M
binlog_format = ROW
innodb_
innodb_
innodb_
innodb_flush_method = O_DIRECT
innodb_
innodb_
wsrep_cluster_name = mycluster
wsrep_cluster_
wsrep_node_name = ip-10-40-197-24
wsrep_provider = /usr/lib64/
wsrep_provider_
wsrep_sst_method = xtrabackup
wsrep_sst_auth = sst:secret
wsrep_replicate
wsrep_slave_threads = 16
wsrep_auto_
innodb_
innodb_
[mysql]
prompt = "ip-10-40-197-24 mysql> "
[client]
user = root
I have not modified /usr/bin/
Related branches
Changed in percona-xtradb-cluster: | |
milestone: | none → 5.5.33-23.7.6 |
Changed in percona-xtradb-cluster: | |
status: | New → In Progress |
assignee: | nobody → Raghavendra D Prabhu (raghavendra-prabhu) |
importance: | Undecided → High |
Changed in percona-xtradb-cluster: | |
status: | In Progress → Fix Committed |
Changed in percona-xtradb-cluster: | |
status: | Fix Committed → In Progress |
summary: |
- SST xtrabackup in 5.5.31 (no tweaking) is not clobbering JOINER datadir + SST xtrabackup in 5.5.31 is not clobbering JOINER datadir |
Changed in percona-xtradb-cluster: | |
status: | Fix Committed → Fix Released |
Is the SST fully complete when it is checked with ls?
Need the innobackupex logs and error logs of donor and joiner.
Since the default streaming is done with tar - assuming same configuration on donor and joiner - then the clobbering should be fine.
Few other things:
a) Note that the timestamp is identical before and after on that node.
" -rw-rw----. 1 mysql mysql 243269632 Aug 16 12:41 /var/lib/ mysql/test/ sbtest1. ibd"
" -rw-rw----. 1 mysql mysql 243269632 Aug 16 12:41 /var/lib/ mysql/test/ sbtest1. ibd"
b) The file sizes are different. (and with a large difference)
-rw-rw----. 1 mysql mysql 214958080 Aug 16 12:41 /var/lib/ mysql/test/ sbtest1. ibd
on first node
v/s
-rw-rw----. 1 mysql mysql 243269632 Aug 16 12:41 /var/lib/ mysql/test/ sbtest1. ibd
on others.
This indicates inconsistency beforehand.
c) Because of point b and/or due to others, SST may not have occurred correctly. Hence the logs are required.