rsync-server from previous IST keeps running between restarts
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC |
New
|
Undecided
|
Unassigned |
Bug Description
root@database-
ii percona-
ii percona-
ii percona-
ii percona-
If a server startup fails immediately after IST-ing the rsync daemon does not get removed , which complicated a next start of the server.
Mar 4 00:14:25 localhost mysqld: 130304 0:14:25 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '172.16.15.4' --auth '' --datadir '/var/lib/mysql/' -
-defaults-file '/etc/mysql/my.cnf' --parent '29769'
Mar 4 00:14:25 localhost mysqld: #011Read: 'rsync daemon already running.'
Perhaps the IST could also be taught to understand "rsync daemon already running"?
summary: |
- rsync-server from SST keeps running between restarts + rsync-server from previous IST keeps running between restarts |
Realizing that this is really short on info, my apologies, this is what happened:
I had a cluster up, ubuntu precise, 3 virtual servers. I was busy importing quite a large dataset (161G), a mysqldump from another server, which made the binlogs fill up the disk (and there was general logging turned on). So the imported failed halfway:
Mar 3 23:15:12 localhost mysqld: 130303 23:15:12 InnoDB: Error: Write to file ./ccpr/ mycc_blob. ibd failed at offset 0 2478833664.
Mar 3 23:15:12 localhost mysqld: InnoDB: 1048576 bytes should have been written, only 0 were written.
Mar 3 23:15:12 localhost mysqld: InnoDB: Operating system error number 28.
So I '/etc/init.d/mysql stop'-ped the one I was importing on, since that was also the one with the .sql files that were being imported. Then I discovered general logging was on. I commented out the general logging, removed the general log and the binlogs, and started up the server again with /etc/init.d/mysql start. The server would not start:
Mar 3 23:47:06 localhost mysqld_safe: WSREP: Running position recovery with --log_error= /tmp/tmp. DUXaE2PwQ3 uuid=4cf3fb18- 8454-11e2- 0800-2ede888bd8 81,start_ prim=0, npvo=0, ignore_ sb=0,ignore_ quorum= 0,state= 1,last_ sent_seq= 0,checksum= 1,instances= 7a9f-11e2- 0800-ab34640715 1b,prim= 1,last_ seq=51994, last_prim= view_id( PRIM,0d56e0a3- 7a9f-11e2- 0800-ab34640715 1b,1),to_ seq=51993, weight= 1 8454-11e2- 0800-2ede888bd8 81,prim= 0,last_ seq=4294967295, last_prim= view_id( NON_PRIM, 00000000- 0000-0000- 0000-0000000000 00,0),to_ seq=-1, weight= 1 7a9b-11e2- 0800-596f0bbb24 4c,prim= 1,last_ seq=2,last_ prim=view_ id(PRIM, 7f1d550f- 7a9b-11e2- 0800-596f0bbb24 4c,13), to_seq= 2852875, weight= 1 version= 0,type= 1,user_ type=255, order=4, seq=0,seq_ range=0, aru_seq= -1,flags= 4,source= 7f1d550f- 7a9b-11e2- 0800-596f0bbb24 4c,source_ view_id= view_id( REG,0d56e0a3- 7a9f-11e2- 0800-ab34640715 1b,14), range_uuid= 00000000- 0000-0000- 0000-0000000000 00,range= [-1,-1] ,fifo_seq= 4778331, node_list= () version= 0,type= 3,user_ type=255, order=1, seq=0,seq_ range=- 1,aru_seq= 0,flags= 4,source= 7f1d550f- 7a9b-11e2- 0800-596f0bbb24 4c,source_ view_id= view_id( REG,0d56e0a3- 7a9f-11e2- 0800-ab34640715 1b,14), range_uuid= 00000000- 0000-0000- 0000-0000000000 00,range= [-1,-1] ,fifo_seq= 4778333, node_list= () evs::proto( 4cf3fb18- 8454-11e2- 0800-2ede888bd8 81, OPERATIONAL, view_id( REG,0d56e0a3- 7a9f-11e2- 0800-ab34640715 1b,14)) , OPERATIONAL) { view=view( view_id( REG,0d56e0a3- 7a9f-11e2- 0800-ab34640715 1b,14) mem...
Mar 3 23:47:17 localhost mysqld_safe: WSREP: Failed to recover position:
[...]
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: caught exception in PC, state dump to stderr follows:
Mar 3 23:47:18 localhost mysqld: pc::Proto{
Mar 3 23:47:18 localhost mysqld: #0110d56e0a3-
Mar 3 23:47:18 localhost mysqld: #0114cf3fb18-
Mar 3 23:47:18 localhost mysqld: #0117f1d550f-
[...]
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [Note] WSREP: evs::msg{
Mar 3 23:47:18 localhost mysqld: } 64
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: exception caused by message: evs::msg{
Mar 3 23:47:18 localhost mysqld: }
Mar 3 23:47:18 localhost mysqld: 130303 23:47:18 [ERROR] WSREP: state after handling message: evs::proto(
Mar 3 23:47:18 localhost mysqld: current_