rsync-server from previous IST keeps running between restarts

Bug #1143052 reported by Mrten
This bug report is a duplicate of:  Bug #1169676: rsync issues with SST/IST on joiner. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
New
Undecided
Unassigned

Bug Description

root@database-3:/var/tmp# dpkg -l | grep cluster
ii percona-xtradb-cluster-client-5.5 5.5.29-23.7.1-387.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.29-23.7.1-387.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 143.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.29-23.7.1-387.precise Percona Server database server binaries

If a server startup fails immediately after IST-ing the rsync daemon does not get removed , which complicated a next start of the server.

Mar 4 00:14:25 localhost mysqld: 130304 0:14:25 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '172.16.15.4' --auth '' --datadir '/var/lib/mysql/' -
-defaults-file '/etc/mysql/my.cnf' --parent '29769'
Mar 4 00:14:25 localhost mysqld: #011Read: 'rsync daemon already running.'

Perhaps the IST could also be taught to understand "rsync daemon already running"?

Mrten (bugzilla-ii)
summary: - rsync-server from SST keeps running between restarts
+ rsync-server from previous IST keeps running between restarts
Revision history for this message
Mrten (bugzilla-ii) wrote :
Download full text (8.5 KiB)

Realizing that this is really short on info, my apologies, this is what happened:

I had a cluster up, ubuntu precise, 3 virtual servers. I was busy importing quite a large dataset (161G), a mysqldump from another server, which made the binlogs fill up the disk (and there was general logging turned on). So the imported failed halfway:

Mar 3 23:15:12 localhost mysqld: 130303 23:15:12 InnoDB: Error: Write to file ./ccpr/mycc_blob.ibd failed at offset 0 2478833664.
Mar 3 23:15:12 localhost mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
Mar 3 23:15:12 localhost mysqld: InnoDB: Operating system error number 28.

So I '/etc/init.d/mysql stop'-ped the one I was importing on, since that was also the one with the .sql files that were being imported. Then I discovered general logging was on. I commented out the general logging, removed the general log and the binlogs, and started up the server again with /etc/init.d/mysql start. The server would not start:

Mar 3 23:47:06 localhost mysqld_safe: WSREP: Running position recovery with --log_error=/tmp/tmp.DUXaE2PwQ3
Mar 3 23:47:17 localhost mysqld_safe: WSREP: Failed to recover position:
[...]
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: caught exception in PC, state dump to stderr follows:
Mar 3 23:47:18 localhost mysqld: pc::Proto{uuid=4cf3fb18-8454-11e2-0800-2ede888bd881,start_prim=0,npvo=0,ignore_sb=0,ignore_quorum=0,state=1,last_sent_seq=0,checksum=1,instances=
Mar 3 23:47:18 localhost mysqld: #0110d56e0a3-7a9f-11e2-0800-ab346407151b,prim=1,last_seq=51994,last_prim=view_id(PRIM,0d56e0a3-7a9f-11e2-0800-ab346407151b,1),to_seq=51993,weight=1
Mar 3 23:47:18 localhost mysqld: #0114cf3fb18-8454-11e2-0800-2ede888bd881,prim=0,last_seq=4294967295,last_prim=view_id(NON_PRIM,00000000-0000-0000-0000-000000000000,0),to_seq=-1,weight=1
Mar 3 23:47:18 localhost mysqld: #0117f1d550f-7a9b-11e2-0800-596f0bbb244c,prim=1,last_seq=2,last_prim=view_id(PRIM,7f1d550f-7a9b-11e2-0800-596f0bbb244c,13),to_seq=2852875,weight=1
[...]
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [Note] WSREP: evs::msg{version=0,type=1,user_type=255,order=4,seq=0,seq_range=0,aru_seq=-1,flags=4,source=7f1d550f-7a9b-11e2-0800-596f0bbb244c,source_view_id=view_id(REG,0d56e0a3-7a9f-11e2-0800-ab346407151b,14),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=4778331,node_list=()
Mar 3 23:47:18 localhost mysqld: } 64
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: exception caused by message: evs::msg{version=0,type=3,user_type=255,order=1,seq=0,seq_range=-1,aru_seq=0,flags=4,source=7f1d550f-7a9b-11e2-0800-596f0bbb244c,source_view_id=view_id(REG,0d56e0a3-7a9f-11e2-0800-ab346407151b,14),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=4778333,node_list=()
Mar 3 23:47:18 localhost mysqld: }
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: state after handling message: evs::proto(evs::proto(4cf3fb18-8454-11e2-0800-2ede888bd881, OPERATIONAL, view_id(REG,0d56e0a3-7a9f-11e2-0800-ab346407151b,14)), OPERATIONAL) {
Mar 3 23:47:18 localhost mysqld: current_view=view(view_id(REG,0d56e0a3-7a9f-11e2-0800-ab346407151b,14) mem...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.