node won't synchronize new data from cluster following an outage on that node

[Brad Fino]

I have a 3 node test cluster comprising nodes A, B, and C.

I brought up Node A first, followed by B and C, and created a couple databases, which are synced across the cluster.

$ dsh -e -g xtradb-cluster.ops "/usr/local/mysql/bin/mysql -e 'show databases'" | grep -v Database | grep -v schema | grep -v mysql
xtradb-cluster101.ops: blah
xtradb-cluster101.ops: tester
xtradb-cluster102.ops: blah
xtradb-cluster102.ops: tester
xtradb-cluster103.ops: blah
xtradb-cluster103.ops: tester

I then cleanly shutdown node A and added a new database on node B, which synchronized to nodes B and C.

$ dsh -e -g xtradb-cluster.ops "/usr/local/mysql/bin/mysql -e 'show databases'" | grep -v Database | grep -v schema | grep -v mysql
xtradb-cluster102.ops: blah
xtradb-cluster102.ops: tester
xtradb-cluster102.ops: this_should_show_up_on_all_3_nodes
xtradb-cluster103.ops: blah
xtradb-cluster103.ops: tester
xtradb-cluster103.ops: this_should_show_up_on_all_3_nodes

I then brought node A back online. I would expect the new database to exist on node A, but it does not.

$ dsh -e -g xtradb-cluster.ops "/usr/local/mysql/bin/mysql -e 'show databases'" | grep -v Database | grep -v schema | grep -v mysql
xtradb-cluster101.ops: blah
xtradb-cluster101.ops: tester
xtradb-cluster102.ops: blah
xtradb-cluster102.ops: tester
xtradb-cluster102.ops: this_should_show_up_on_all_3_nodes
xtradb-cluster103.ops: blah
xtradb-cluster103.ops: tester
xtradb-cluster103.ops: this_should_show_up_on_all_3_nodes

The cluster is still in sync, somewhat, as I'm able to issue "drop database this_should_show_up_on_all_3_nodes" on node A, which then deletes the database on nodes B and C.

Here are the relevant lines in my.cnf on all 3 nodes. The only difference is on node A where gcomm:// is empty.

wsrep_provider=/usr/local/mysql/lib/libgalera_smm.so
wsrep_cluster_address=gcomm://10.202.1.37
wsrep_slave_threads=8
wsrep_sst_method=rsync
wsrep_cluster_name=SuperAwesomeFusionTestCluster
binlog_format=ROW
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1

[Vadim Tkachenko (vadim-tk)]

Do you still have this problem with new 5.5.28 release ?

[Brad Fino (mrfino)]

I haven't upgrade to 5.5.28 yet ... but I can certainly run this through again and see if I can reproduce it.

[Brad Fino (mrfino)]

Hi,

Just tried again with the generic Linux binary version of Percona-XtraDB-Cluster-5.5.28-23.7.373.Linux.x86_64 on RHEL6.

Still has the same issue.

Note the absence of 'test4' database on 101, which was offline while test4 was created on 102.

mysql-clusterdb101.prod: test1
mysql-clusterdb101.prod: test2
mysql-clusterdb101.prod: test3

mysql-clusterdb102.prod: test1
mysql-clusterdb102.prod: test2
mysql-clusterdb102.prod: test3
mysql-clusterdb102.prod: test4

mysql-clusterdb103.prod: test1
mysql-clusterdb103.prod: test2
mysql-clusterdb103.prod: test3
mysql-clusterdb103.prod: test4

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Brad,

Can you provide logs of A?

[Krunal Bauskar (krunal-bauskar)]

I tried this with latest PXC trunk

1. Started node-1,2,3
2. node-1: Created table t1, t2, t3
3. node-2: show tables (got t1,t2, t3)
4. node-3: show tables (got t1, t2, t3)
5. shutdown node-2
6. node-1: create table t4, drop table t2
7. node-3: show tables (got t1, t3, t4)
8. re-start node-2
9. node-2: show tables (got t1, t3, t4) .... so node-2 on restart is in sync.
10. node-2: drop table t1 (reflected on node-1 and 3 as expected)
11. node-2: drop table t4 (reflected on node-1 and 3 as expected)
12. node-3: drop table t3 (reflected on node-1 and 2 as expected)

---------------

So with recent PXC trunk things are working fine.
Please try to upgrade to recent version.
(Bug is almost 3 yrs old may be something was missing then but upgrading to recent version is good)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1218