Thanks Johan!

On 11/07/2014 12:00 PM, johan wrote:
> Hi,
>
> Here is a trace:
>
> http://paste.ubuntu.com/8420587/
>
> Teemu@codership then said that "thread #28 seems suspicious".
>
> Then no more info on this.
> However, since upgrading to (done  by David on one node) I have never seen
> this issue.
> Despite 100s of restarts.
>
> Now i have update the all nodes to use:
>
> Percona-XtraDB-Cluster-server-56-5.6.21-25.8.910.el6.x86_64
> Percona-XtraDB-Cluster-galera-3-3.8-1.3323.rhel6.x86_64
>
> And as i said, I have never seen the issue again.
>
> I also have this version installed, on all nodes.
>
> percona-xtrabackup-2.2.6-5042.el6.x86_64
>
> I assume something was changed was changed in the problem area in one or
> more of the version mentioned above
>
> BR
> johan
>
>
>
> On Fri, Nov 7, 2014 at 6:23 PM, Raghavendra D Prabhu <
> <email address hidden>> wrote:
>
>> Make sure debug symbol packages are installed before attaching gdb for
>> that.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1373796
>>
>> Title:
>>   SST: Joiner stuck in 'joining', but error log reports 'synced'
>>
>> Status in MySQL patches by Codership:
>>   New
>> Status in MySQL patches by Codership 5.5 series:
>>   New
>> Status in MySQL patches by Codership 5.6 series:
>>   New
>> Status in Percona XtraDB Cluster - HA scalable solution for MySQL:
>>   New
>> Status in Percona XtraDB Cluster 5.5 series:
>>   New
>> Status in Percona XtraDB Cluster 5.6 series:
>>   New
>>
>> Bug description:
>>   Environment:
>>   * PXC  5.6.20-68.0-56
>>   * Galera provider:  3.7(r7f44a18)
>>   * xtrabackup 2.2.4
>>   * EC2, m3.medium, ubuntu-trusty-14.04-amd64-server-20140607.1
>> (ami-0307d674)
>>
>>   Galera nodes, all are 'synced':
>>   10.0.0.24
>>   10.0.0.25
>>   10.0.0.26
>>
>>   ssh 10.0.0.24
>>   sudo killall -9  mysqld mysqld_safe
>>   # remove mysql data files to have an SST:
>>   sudo rm -rf /mnt/data/mysql/*
>>   sudo mysqld_safe &
>>
>>   10.0.0.24 is now the JOINER and writes this in the mysql error log:
>>
>>   140925 07:53:55 mysqld_safe Starting mysqld daemon with databases from
>> /mnt/data/mysql/
>>   140925 07:53:55 mysqld_safe Skipping wsrep-recover for empty datadir:
>> /mnt/data/mysql/
>>   140925 07:53:55 mysqld_safe Assigning
>> 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
>>   2014-09-25 07:53:56 0 [Note] WSREP: wsrep_start_position var submitted:
>> '00000000-0000-0000-0000-000000000000:-1'
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Read nil XID from storage
>> engines, skipping position init
>>   2014-09-25 07:53:56 28987 [Note] WSREP: wsrep_load(): loading provider
>> library '/usr/lib/libgalera_smm.so'
>>   2014-09-25 07:53:56 28987 [Note] WSREP: wsrep_load(): Galera
>> 3.7(r7f44a18) by Codership Oy <email address hidden> loaded successfully.
>>   2014-09-25 07:53:56 28987 [Note] WSREP: CRC-32C: using hardware
>> acceleration.
>>   2014-09-25 07:53:56 28987 [Warning] WSREP: Could not open saved state
>> file for reading: /mnt/data/mysql//grastate.dat
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Found saved state:
>> 00000000-0000-0000-0000-000000000000:-1
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Passing config to GCS: base_host
>> = 10.0.0.24; base_port = 4567; cert.log_conflicts = no; debug = no;
>> evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S;
>> evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3;
>> evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout =
>> PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir
>> = /mnt/data/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0;
>> gcache.name = /mnt/data/mysql//galera.cache; gcache.page_size = 128M;
>> gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit =
>> 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle
>> = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit
>> = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0;
>> pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false;
>> pc.ignore_sb = false; pc.npvo = false; pc.recovery = 1; pc.version = 0;
>> pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.wei
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Assign initial position for
>> certification: -1, protocol version: -1
>>   2014-09-25 07:53:56 28987 [Note] WSREP: wsrep_sst_grab()
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Start replication
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Setting initial position to
>> 00000000-0000-0000-0000-000000000000:-1
>>   2014-09-25 07:53:56 28987 [Note] WSREP: protonet asio version 0
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Using CRC-32C for message
>> checksums.
>>   2014-09-25 07:53:56 28987 [Note] WSREP: backend: asio
>>   2014-09-25 07:53:56 28987 [Warning] WSREP: access file(gvwstate.dat)
>> failed(No such file or directory)
>>   2014-09-25 07:53:56 28987 [Note] WSREP: restore pc from disk failed
>>   2014-09-25 07:53:56 28987 [Note] WSREP: GMCast version 0
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> listening at tcp://0.0.0.0:4567
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> multicast: , ttl: 1
>>   2014-09-25 07:53:56 28987 [Note] WSREP: EVS version 0
>>   2014-09-25 07:53:56 28987 [Note] WSREP: PC version 0
>>   2014-09-25 07:53:56 28987 [Note] WSREP: gcomm: connecting to group
>> 'my_wsrep_cluster', peer '10.0.0.25:4567,10.0.0.26:4567,10.0.0.24:4567'
>>   2014-09-25 07:53:56 28987 [Warning] WSREP: (1a021a43, 'tcp://
>> 0.0.0.0:4567') address 'tcp://10.0.0.24:4567' points to own listening
>> address, blacklisting
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.24:4567' pointing to uuid 1a021a43 is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> turning message relay requesting on, nonlive peers:
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.24:4567' pointing to uuid 1a021a43 is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.24:4567' pointing to uuid 1a021a43 is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 28987 [Note] WSREP: declaring 1076e92e at tcp://
>> 10.0.0.25:4567 stable
>>   2014-09-25 07:53:56 28987 [Note] WSREP: declaring 35b8b436 at tcp://
>> 10.0.0.26:4567 stable
>>   2014-09-25 07:53:56 28987 [Note] WSREP: Node 1076e92e state prim
>>   2014-09-25 07:53:56 28987 [Note] WSREP: save pc into disk
>>   2014-09-25 07:53:57 28987 [Note] WSREP: gcomm: connected
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Changing maximum packet size to
>> 64500, resulting msg size: 32636
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Opened channel 'my_wsrep_cluster'
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Waiting for SST to complete.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: New COMPONENT: primary = yes,
>> bootstrap = no, my_idx = 1, memb_num = 3
>>   2014-09-25 07:53:57 28987 [Note] WSREP: STATE EXCHANGE: Waiting for
>> state UUID.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: STATE EXCHANGE: sent state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e
>>   2014-09-25 07:53:57 28987 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 0 (10.0.0.25)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 2 (10.0.0.26)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 1 (10.0.0.24)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Quorum results:
>>         version    = 3,
>>         component  = PRIMARY,
>>         conf_id    = 20,
>>         members    = 2/3 (joined/total),
>>         act_id     = 10,
>>         last_appl. = -1,
>>         protocols  = 0/6/3 (gcs/repl/appl),
>>         group UUID = f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Flow-control interval: [28, 28]
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 10)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: State transfer required:
>>         Group state: f7d5ff68-4482-11e4-9f6c-5f76e5087b34:10
>>         Local state: 00000000-0000-0000-0000-000000000000:-1
>>   2014-09-25 07:53:57 28987 [Note] WSREP: New cluster view: global state:
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:10, view# 21: Primary, number of
>> nodes: 3, my index: 1, protocol version 3
>>   2014-09-25 07:53:57 28987 [Warning] WSREP: Gap in state sequence. Need
>> state transfer.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Running:
>> 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.24' --auth
>> 'root:password' --datadir '/mnt/data/mysql/' --defaults-file
>> '/etc/mysql/my.cnf' --parent '28987'  '' '
>>   WSREP_SST: [INFO] Streaming with xbstream (20140925 07:53:57.315)
>>   WSREP_SST: [INFO] Using socat as streamer (20140925 07:53:57.329)
>>   WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u
>> TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} )
>> (20140925 07:53:57.799)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Prepared SST request:
>> xtrabackup-v2|10.0.0.24:4444/xtrabackup_sst
>>   2014-09-25 07:53:57 28987 [Note] WSREP: wsrep_notify_cmd is not defined,
>> skipping notification.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: REPL Protocols: 6 (3, 2)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Assign initial position for
>> certification: 10, protocol version: 3
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:57 28987 [Warning] WSREP: Failed to prepare for
>> incremental state transfer: Local state UUID
>> (00000000-0000-0000-0000-000000000000) does not match group state UUID
>> (f7d5ff68-4482-11e4-9f6c-5f76e5087b34): 1 (Operation not permitted)
>>          at galera/src/replicator_str.cpp:prepare_for_IST():455. IST will
>> be unavailable.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Member 1.0 (10.0.0.24) requested
>> state transfer from '*any*'. Selected 0.0 (10.0.0.25)(SYNCED) as donor.
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Shifting PRIMARY -> JOINER (TO:
>> 10)
>>   2014-09-25 07:53:57 28987 [Note] WSREP: Requesting state transfer:
>> success, donor: 0
>>   WSREP_SST: [INFO] Proceeding with SST (20140925 07:53:58.763)
>>   WSREP_SST: [INFO] Cleaning the existing datadir (20140925 07:53:58.765)
>>   removed ‘/mnt/data/mysql/gvwstate.dat’
>>   WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio |
>> xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140925 07:53:58.782)
>>   2014-09-25 07:53:59 28987 [Note] WSREP: (1a021a43, 'tcp://0.0.0.0:4567')
>> turning message relay requesting off
>>   2014-09-25 07:54:12 28987 [Note] WSREP: 0.0 (10.0.0.25): State transfer
>> to 1.0 (10.0.0.24) complete.
>>   2014-09-25 07:54:12 28987 [Note] WSREP: Member 0.0 (10.0.0.25) synced
>> with group.
>>   WSREP_SST: [INFO] Preparing the backup at /mnt/data/mysql/ (20140925
>> 07:54:12.694)
>>   WSREP_SST: [INFO] Evaluating innobackupex --no-version-check
>> --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20140925
>> 07:54:12.696)
>>   WSREP_SST: [INFO] Total time on joiner: 0 seconds (20140925 07:54:23.226)
>>   WSREP_SST: [INFO] Removing the sst_in_progress file (20140925
>> 07:54:23.229)
>>   2014-09-25 07:54:23 28987 [Note] WSREP: SST complete, seqno: 7
>>   2014-09-25 07:54:23 28987 [Note] Plugin 'FEDERATED' is disabled.
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Using atomics to ref count
>> buffer pool pages
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: The InnoDB memory heap is
>> disabled
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Mutexes and rw_locks use GCC
>> atomic builtins
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Memory barrier is not used
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Compressed tables use zlib 1.2.8
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Using Linux native AIO
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Using CPU crc32 instructions
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Initializing buffer pool, size
>> = 2.4G
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Completed initialization of
>> buffer pool
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Highest supported file format
>> is Barracuda.
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: 128 rollback segment(s) are
>> active.
>>   2014-09-25 07:54:23 28987 [Note] InnoDB: Waiting for purge to start
>>   2014-09-25 07:54:23 28987 [Note] InnoDB:  Percona XtraDB (
>> http://www.percona.com) 5.6.20-68.0 started; log sequence number 1627158
>>   2014-09-25 07:54:23 28987 [Warning] No existing UUID has been found, so
>> we assume that this is the first time that this server has been started.
>> Generating a new UUID: 2a2ea9d3-4489-11e4-a0f4-02bb8ea7691b.
>>   2014-09-25 07:54:23 28987 [Note] RSA private key file not found:
>> /mnt/data/mysql//private_key.pem. Some authentication plugins will not work.
>>   2014-09-25 07:54:23 28987 [Note] RSA public key file not found:
>> /mnt/data/mysql//public_key.pem. Some authentication plugins will not work.
>>   2014-09-25 07:54:23 28987 [Note] Server hostname (bind-address): '*';
>> port: 3306
>>   2014-09-25 07:54:23 28987 [Note] IPv6 is available.
>>   2014-09-25 07:54:23 28987 [Note]   - '::' resolves to '::';
>>   2014-09-25 07:54:23 28987 [Note] Server socket created on IP: '::'.
>>   2014-09-25 07:54:23 28987 [Warning] 'proxies_priv' entry '@
>> root@ip-10-0-0-24' ignored in --skip-name-resolve mode.
>>   2014-09-25 07:54:23 28987 [Note] Event Scheduler: Loaded 0 events
>>   2014-09-25 07:54:23 28987 [Note] WSREP: Signalling provider to continue.
>>   2014-09-25 07:54:23 28987 [Note] WSREP: inited wsrep sidno 1
>>   2014-09-25 07:54:23 28987 [Note] WSREP: SST received:
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:7
>>   2014-09-25 07:54:23 28987 [Note] /usr/sbin/mysqld: ready for connections.
>>   Version: '5.6.20-68.0-56'  socket: '/var/run/mysqld/mysqld.sock'  port:
>> 3306  Percona XtraDB Cluster (GPL), Release 25.7, wsrep_25.7.r4126
>>   2014-09-25 07:54:23 28987 [Note] WSREP: 1.0 (10.0.0.24): State transfer
>> from 0.0 (10.0.0.25) complete.
>>   2014-09-25 07:54:23 28987 [Note] WSREP: Shifting JOINER -> JOINED (TO:
>> 10)
>>   2014-09-25 07:54:23 28987 [Note] WSREP: Member 1.0 (10.0.0.24) synced
>> with group.
>>   2014-09-25 07:54:23 28987 [Note] WSREP: Shifting JOINED -> SYNCED (TO:
>> 10)
>>
>>
>>   mysql> SHOW GLOBAL STATUS LIKE 'wsrep%';
>>
>> +------------------------------+----------------------------------------------+
>>   | Variable_name                | Value
>>       |
>>
>> +------------------------------+----------------------------------------------+
>>   | wsrep_local_state_uuid       | f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>      |
>>   | wsrep_protocol_version       | 6
>>       |
>>   | wsrep_last_committed         | 7
>>       |
>>   | wsrep_replicated             | 0
>>       |
>>   | wsrep_replicated_bytes       | 0
>>       |
>>   | wsrep_repl_keys              | 0
>>       |
>>   | wsrep_repl_keys_bytes        | 0
>>       |
>>   | wsrep_repl_data_bytes        | 0
>>       |
>>   | wsrep_repl_other_bytes       | 0
>>       |
>>   | wsrep_received               | 3
>>       |
>>   | wsrep_received_bytes         | 290
>>       |
>>   | wsrep_local_commits          | 0
>>       |
>>   | wsrep_local_cert_failures    | 0
>>       |
>>   | wsrep_local_replays          | 0
>>       |
>>   | wsrep_local_send_queue       | 0
>>       |
>>   | wsrep_local_send_queue_max   | 1
>>       |
>>   | wsrep_local_send_queue_min   | 0
>>       |
>>   | wsrep_local_send_queue_avg   | 0.000000
>>      |
>>   | wsrep_local_recv_queue       | 0
>>       |
>>   | wsrep_local_recv_queue_max   | 1
>>       |
>>   | wsrep_local_recv_queue_min   | 0
>>       |
>>   | wsrep_local_recv_queue_avg   | 0.000000
>>      |
>>   | wsrep_local_cached_downto    | 18446744073709551615
>>      |
>>   | wsrep_flow_control_paused_ns | 0
>>       |
>>   | wsrep_flow_control_paused    | 0.000000
>>      |
>>   | wsrep_flow_control_sent      | 0
>>       |
>>   | wsrep_flow_control_recv      | 0
>>       |
>>   | wsrep_cert_deps_distance     | 0.000000
>>      |
>>   | wsrep_apply_oooe             | 0.000000
>>      |
>>   | wsrep_apply_oool             | 0.000000
>>      |
>>   | wsrep_apply_window           | 0.000000
>>      |
>>   | wsrep_commit_oooe            | 0.000000
>>      |
>>   | wsrep_commit_oool            | 0.000000
>>      |
>>   | wsrep_commit_window          | 0.000000
>>      |
>>   | wsrep_local_state            | 1
>>       |
>>   | wsrep_local_state_comment    | Joining
>>       |
>>   | wsrep_cert_index_size        | 0
>>       |
>>   | wsrep_causal_reads           | 0
>>       |
>>   | wsrep_cert_interval          | 0.000000
>>      |
>>   | wsrep_incoming_addresses     | 10.0.0.25:3306,10.0.0.24:3306,
>> 10.0.0.26:3306 |
>>   | wsrep_evs_repl_latency       | 0/0/0/0/0
>>       |
>>   | wsrep_cluster_conf_id        | 21
>>      |
>>   | wsrep_cluster_size           | 3
>>       |
>>   | wsrep_cluster_state_uuid     | f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>      |
>>   | wsrep_cluster_status         | Primary
>>       |
>>   | wsrep_connected              | ON
>>      |
>>   | wsrep_local_bf_aborts        | 0
>>       |
>>   | wsrep_local_index            | 1
>>       |
>>   | wsrep_provider_name          | Galera
>>      |
>>   | wsrep_provider_vendor        | Codership Oy <email address hidden>
>>       |
>>   | wsrep_provider_version       | 3.7(r7f44a18)
>>       |
>>   | wsrep_ready                  | OFF
>>       |
>>
>> +------------------------------+----------------------------------------------+
>>   52 rows in set (0.00 sec)
>>
>>   **** There are no traces of xtrabackup still running:
>>
>>   ubuntu@ip-10-0-0-24:~$ ps -ef |grep inno
>>   ubuntu   29349 28112  0 07:55 pts/0    00:00:00 grep --color=auto inno
>>   ubuntu@ip-10-0-0-24:~$ ps -ef |grep mysqld
>>   root     28143 28112  0 07:53 pts/0    00:00:00 sudo mysqld_safe
>>   root     28144 28143  0 07:53 pts/0    00:00:00 /bin/sh
>> /usr/bin/mysqld_safe
>>   mysql    28987 28144  1 07:53 pts/0    00:00:01 /usr/sbin/mysqld
>> --basedir=/usr/ --datadir=/mnt/data/mysql/
>> --plugin-dir=/usr//lib/mysql/plugin --user=mysql
>> --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/log/mysql.log
>> --pid-file=/mnt/data/mysql//mysqld.pid --socket=/var/run/mysqld/mysqld.sock
>> --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
>>
>>   **** Looking at the innobackup prepare log:
>>
>>   ubuntu@ip-10-0-0-24:~$ sudo cat /mnt/data/mysql/innobackup.prepare.log
>>   sudo: unable to resolve host ip-10-0-0-24
>>
>>   InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
>>   and Percona LLC and/or its affiliates 2009-2013.  All Rights Reserved.
>>
>>   This software is published under
>>   the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
>>
>>   Get the latest version of Percona XtraBackup, documentation, and help
>> resources:
>>   http://www.percona.com/xb/p
>>
>>   IMPORTANT: Please check that the apply-log run completes successfully.
>>              At the end of a successful apply-log run innobackupex
>>              prints "completed OK!".
>>
>>
>>   140925 07:54:13  innobackupex: Starting ibbackup with command:
>>   xtrabackup  --defaults-file="/mnt/data/mysql/backup-my.cnf"
>>   --defaults-group="mysqld" --prepare --target-dir=/mnt/data/mysql
>>   --tmpdir=/tmp
>>
>>   xtrabackup version 2.2.4 based on MySQL server 5.6.17 Linux (x86_64)
>> (revision id: )
>>   xtrabackup: cd to /mnt/data/mysql
>>   xtrabackup: This target seems to be not prepared yet.
>>   xtrabackup: xtrabackup_logfile detected: size=2097152,
>> start_lsn=(1627118)
>>   xtrabackup: using the following InnoDB configuration for recovery:
>>   xtrabackup:   innodb_data_home_dir = ./
>>   xtrabackup:   innodb_data_file_path = ibdata1:100M:autoextend
>>   xtrabackup:   innodb_log_group_home_dir = ./
>>   xtrabackup:   innodb_log_files_in_group = 1
>>   xtrabackup:   innodb_log_file_size = 2097152
>>   xtrabackup: using the following InnoDB configuration for recovery:
>>   xtrabackup:   innodb_data_home_dir = ./
>>   xtrabackup:   innodb_data_file_path = ibdata1:100M:autoextend
>>   xtrabackup:   innodb_log_group_home_dir = ./
>>   xtrabackup:   innodb_log_files_in_group = 1
>>   xtrabackup:   innodb_log_file_size = 2097152
>>   xtrabackup: Starting InnoDB instance for recovery.
>>   xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory
>> parameter)
>>   InnoDB: Using atomics to ref count buffer pool pages
>>   InnoDB: The InnoDB memory heap is disabled
>>   InnoDB: Mutexes and rw_locks use GCC atomic builtins
>>   InnoDB: Compressed tables use zlib 1.2.8
>>   InnoDB: Using CPU crc32 instructions
>>   InnoDB: Initializing buffer pool, size = 100.0M
>>   InnoDB: Completed initialization of buffer pool
>>   InnoDB: Highest supported file format is Barracuda.
>>   InnoDB: The log sequence numbers 1626646 and 1626646 in ibdata files do
>> not match the log sequence number 1627118 in the ib_logfiles!
>>   InnoDB: Database was not shutdown normally!
>>   InnoDB: Starting crash recovery.
>>   InnoDB: Reading tablespace information from the .ibd files...
>>   InnoDB: Restoring possible half-written data pages
>>   InnoDB: from the doublewrite buffer...
>>   InnoDB: 128 rollback segment(s) are active.
>>   InnoDB: Waiting for purge to start
>>   InnoDB: 5.6.17 started; log sequence number 1627118
>>
>>   [notice (again)]
>>     If you use binary log and don't use any hack of group commit,
>>     the binary log position seems to be:
>>
>>   xtrabackup: Recovered WSREP position:
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:7
>>   xtrabackup: starting shutdown with innodb_fast_shutdown = 1
>>   InnoDB: FTS optimize thread exiting.
>>   InnoDB: Starting shutdown...
>>   InnoDB: Shutdown completed; log sequence number 1627128
>>
>>   140925 07:54:14  innobackupex: Restarting xtrabackup with command:
>> xtrabackup  --defaults-file="/mnt/data/mysql/backup-my.cnf"
>> --defaults-group="mysqld" --prepare --target-dir=/mnt/data/mysql
>> --tmpdir=/tmp
>>   for creating ib_logfile*
>>
>>   xtrabackup version 2.2.4 based on MySQL server 5.6.17 Linux (x86_64)
>> (revision id: )
>>   xtrabackup: cd to /mnt/data/mysql
>>   xtrabackup: This target seems to be already prepared.
>>   xtrabackup: notice: xtrabackup_logfile was already used to '--prepare'.
>>   xtrabackup: using the following InnoDB configuration for recovery:
>>   xtrabackup:   innodb_data_home_dir = ./
>>   xtrabackup:   innodb_data_file_path = ibdata1:100M:autoextend
>>   xtrabackup:   innodb_log_group_home_dir = ./
>>   xtrabackup:   innodb_log_files_in_group = 2
>>   xtrabackup:   innodb_log_file_size = 536870912
>>   xtrabackup: using the following InnoDB configuration for recovery:
>>   xtrabackup:   innodb_data_home_dir = ./
>>   xtrabackup:   innodb_data_file_path = ibdata1:100M:autoextend
>>   xtrabackup:   innodb_log_group_home_dir = ./
>>   xtrabackup:   innodb_log_files_in_group = 2
>>   xtrabackup:   innodb_log_file_size = 536870912
>>   xtrabackup: Starting InnoDB instance for recovery.
>>   xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory
>> parameter)
>>   InnoDB: Using atomics to ref count buffer pool pages
>>   InnoDB: The InnoDB memory heap is disabled
>>   InnoDB: Mutexes and rw_locks use GCC atomic builtins
>>   InnoDB: Compressed tables use zlib 1.2.8
>>   InnoDB: Using CPU crc32 instructions
>>   InnoDB: Initializing buffer pool, size = 100.0M
>>   InnoDB: Completed initialization of buffer pool
>>   InnoDB: Setting log file ./ib_logfile101 size to 512 MB
>>   InnoDB: Progress in MB: 100 200 300 400 500
>>   InnoDB: Setting log file ./ib_logfile1 size to 512 MB
>>   InnoDB: Progress in MB: 100 200 300 400 500
>>   InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
>>   InnoDB: New log files created, LSN=1627128
>>   InnoDB: Highest supported file format is Barracuda.
>>   InnoDB: 128 rollback segment(s) are active.
>>   InnoDB: Waiting for purge to start
>>   InnoDB: 5.6.17 started; log sequence number 1627148
>>
>>   [notice (again)]
>>     If you use binary log and don't use any hack of group commit,
>>     the binary log position seems to be:
>>
>>   xtrabackup: Recovered WSREP position:
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:7
>>   xtrabackup: starting shutdown with innodb_fast_shutdown = 1
>>   InnoDB: FTS optimize thread exiting.
>>   InnoDB: Starting shutdown...
>>   InnoDB: Shutdown completed; log sequence number 1627158
>>   140925 07:54:23  innobackupex: completed OK!
>>
>>   **** Looking at the DONOR (i hope i have got the most parts of the error
>> log):
>>   2014-09-25 07:53:47 10525 [Note] WSREP: wsrep_notify_cmd is not defined,
>> skipping notification.
>>   2014-09-25 07:53:47 10525 [Note] WSREP: REPL Protocols: 6 (3, 2)
>>   2014-09-25 07:53:47 10525 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:47 10525 [Note] WSREP: Assign initial position for
>> certification: 10, protocol version: 3
>>   2014-09-25 07:53:47 10525 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:50 10525 [Note] WSREP:  cleaning up 9aced1b3 (tcp://
>> 10.0.0.24:4567)
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> turning message relay requesting on, nonlive peers:
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> address 'tcp://10.0.0.25:4567' pointing to uuid 1076e92e is blacklisted,
>> skipping
>>   2014-09-25 07:53:56 10525 [Note] WSREP: declaring 1a021a43 at tcp://
>> 10.0.0.24:4567 stable
>>   2014-09-25 07:53:56 10525 [Note] WSREP: declaring 35b8b436 at tcp://
>> 10.0.0.26:4567 stable
>>   2014-09-25 07:53:56 10525 [Note] WSREP: Node 1076e92e state prim
>>   2014-09-25 07:53:56 10525 [Note] WSREP: save pc into disk
>>   2014-09-25 07:53:56 10525 [Note] WSREP: New COMPONENT: primary = yes,
>> bootstrap = no, my_idx = 0, memb_num = 3
>>   2014-09-25 07:53:56 10525 [Note] WSREP: STATE_EXCHANGE: sent state UUID:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e
>>   2014-09-25 07:53:56 10525 [Note] WSREP: STATE EXCHANGE: sent state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e
>>   2014-09-25 07:53:56 10525 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 0 (10.0.0.25)
>>   2014-09-25 07:53:56 10525 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 2 (10.0.0.26)
>>   2014-09-25 07:53:57 10525 [Note] WSREP: STATE EXCHANGE: got state msg:
>> 1a372f69-4489-11e4-9d92-c2831d253d7e from 1 (10.0.0.24)
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Quorum results:
>>         version    = 3,
>>         component  = PRIMARY,
>>         conf_id    = 20,
>>         members    = 2/3 (joined/total),
>>         act_id     = 10,
>>         last_appl. = 0,
>>         protocols  = 0/6/3 (gcs/repl/appl),
>>         group UUID = f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Flow-control interval: [28, 28]
>>   2014-09-25 07:53:57 10525 [Note] WSREP: New cluster view: global state:
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:10, view# 21: Primary, number of
>> nodes: 3, my index: 0, protocol version 3
>>   2014-09-25 07:53:57 10525 [Note] WSREP: wsrep_notify_cmd is not defined,
>> skipping notification.
>>   2014-09-25 07:53:57 10525 [Note] WSREP: REPL Protocols: 6 (3, 2)
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Assign initial position for
>> certification: 10, protocol version: 3
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Service thread queue flushed.
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Member 1.0 (10.0.0.24) requested
>> state transfer from '*any*'. Selected 0.0 (10.0.0.25)(SYNCED) as donor.
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Shifting SYNCED ->
>> DONOR/DESYNCED (TO: 10)
>>   2014-09-25 07:53:57 10525 [Note] WSREP: wsrep_notify_cmd is not defined,
>> skipping notification.
>>   2014-09-25 07:53:57 10525 [Note] WSREP: Running:
>> 'wsrep_sst_xtrabackup-v2 --role 'donor' --address '
>> 10.0.0.24:4444/xtrabackup_sst' --auth 'root:password' --socket
>> '/var/run/mysqld/mysqld.sock' --datadir '/mnt/data/mysql/' --defaults-file
>> '/etc/mysql/my.cnf'   '' --gtid 'f7d5ff68-4482-11e4-9f6c-5f76e5087b34:10''
>>   2014-09-25 07:53:57 10525 [Note] WSREP: sst_donor_thread signaled with 0
>>   WSREP_SST: [INFO] Streaming with xbstream (20140925 07:53:58.240)
>>   WSREP_SST: [INFO] Using socat as streamer (20140925 07:53:58.250)
>>   WSREP_SST: [INFO] Using /tmp/tmp.nnC6BnypNz as xtrabackup temporary
>> directory (20140925 07:53:58.702)
>>   WSREP_SST: [INFO] Using /tmp/tmp.o9h80diUhb as innobackupex temporary
>> directory (20140925 07:53:58.704)
>>   WSREP_SST: [INFO] Streaming GTID file before SST (20140925 07:53:58.716)
>>   WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio
>> TCP:10.0.0.24:4444; RC=( ${PIPESTATUS[@]} ) (20140925 07:53:58.718)
>>   WSREP_SST: [INFO] Sleeping before data transfer for SST (20140925
>> 07:53:58.723)
>>   2014-09-25 07:53:59 10525 [Note] WSREP: (1076e92e, 'tcp://0.0.0.0:4567')
>> turning message relay requesting off
>>   WSREP_SST: [INFO] Streaming the backup to joiner at 10.0.0.24 4444
>> (20140925 07:54:08.734)
>>   WSREP_SST: [INFO] Evaluating innobackupex
>> --defaults-file=/etc/mysql/my.cnf --no-version-check  $tmpopts $INNOEXTRA
>> --galera-info --stream=$sfmt $itmpdir 2>${DATA}/innobackup.backup.log |
>> socat -u stdio TCP:10.0.0.24:4444; RC=( ${PIPESTATUS[@]} ) (20140925
>> 07:54:08.736)
>>   2014-09-25 07:54:11 10525 [Note] WSREP: Provider paused at
>> f7d5ff68-4482-11e4-9f6c-5f76e5087b34:10 (70)
>>   2014-09-25 07:54:12 10525 [Note] WSREP: resuming provider at 70
>>   2014-09-25 07:54:12 10525 [Note] WSREP: Provider resumed.
>>   2014-09-25 07:54:12 10525 [Note] WSREP: 0.0 (10.0.0.25): State transfer
>> to 1.0 (10.0.0.24) complete.
>>   2014-09-25 07:54:12 10525 [Note] WSREP: Shifting DONOR/DESYNCED ->
>> JOINED (TO: 10)
>>   2014-09-25 07:54:12 10525 [Note] WSREP: Member 0.0 (10.0.0.25) synced
>> with group.
>>   2014-09-25 07:54:12 10525 [Note] WSREP: Shifting JOINED -> SYNCED (TO:
>> 10)
>>   2014-09-25 07:54:12 10525 [Note] WSREP: Synchronized with group, ready
>> for connections
>>   2014-09-25 07:54:12 10525 [Note] WSREP: wsrep_notify_cmd is not defined,
>> skipping notification.
>>   WSREP_SST: [INFO] Total time on donor: 0 seconds (20140925 07:54:12.670)
>>   WSREP_SST: [INFO] Cleaning up temporary directories (20140925
>> 07:54:12.672)
>>   2014-09-25 07:54:23 10525 [Note] WSREP: 1.0 (10.0.0.24): State transfer
>> from 0.0 (10.0.0.25) complete.
>>   2014-09-25 07:54:23 10525 [Note] WSREP: Member 1.0 (10.0.0.24) synced
>> with group.
>>
>>
>>   mysql> show global status like 'wsrep%';
>>
>> +------------------------------+----------------------------------------------+
>>   | Variable_name                | Value
>>       |
>>
>> +------------------------------+----------------------------------------------+
>>   | wsrep_local_state_uuid       | f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>      |
>>   | wsrep_protocol_version       | 6
>>       |
>>   | wsrep_last_committed         | 10
>>      |
>>   | wsrep_replicated             | 3
>>       |
>>   | wsrep_replicated_bytes       | 651
>>       |
>>   | wsrep_repl_keys              | 5
>>       |
>>   | wsrep_repl_keys_bytes        | 109
>>       |
>>   | wsrep_repl_data_bytes        | 350
>>       |
>>   | wsrep_repl_other_bytes       | 0
>>       |
>>   | wsrep_received               | 58
>>      |
>>   | wsrep_received_bytes         | 6809
>>      |
>>   | wsrep_local_commits          | 0
>>       |
>>   | wsrep_local_cert_failures    | 0
>>       |
>>   | wsrep_local_replays          | 0
>>       |
>>   | wsrep_local_send_queue       | 0
>>       |
>>   | wsrep_local_send_queue_max   | 1
>>       |
>>   | wsrep_local_send_queue_min   | 0
>>       |
>>   | wsrep_local_send_queue_avg   | 0.000000
>>      |
>>   | wsrep_local_recv_queue       | 0
>>       |
>>   | wsrep_local_recv_queue_max   | 1
>>       |
>>   | wsrep_local_recv_queue_min   | 0
>>       |
>>   | wsrep_local_recv_queue_avg   | 0.000000
>>      |
>>   | wsrep_local_cached_downto    | 2
>>       |
>>   | wsrep_flow_control_paused_ns | 0
>>       |
>>   | wsrep_flow_control_paused    | 0.000000
>>      |
>>   | wsrep_flow_control_sent      | 0
>>       |
>>   | wsrep_flow_control_recv      | 0
>>       |
>>   | wsrep_cert_deps_distance     | 1.000000
>>      |
>>   | wsrep_apply_oooe             | 0.000000
>>      |
>>   | wsrep_apply_oool             | 0.000000
>>      |
>>   | wsrep_apply_window           | 1.000000
>>      |
>>   | wsrep_commit_oooe            | 0.000000
>>      |
>>   | wsrep_commit_oool            | 0.000000
>>      |
>>   | wsrep_commit_window          | 1.000000
>>      |
>>   | wsrep_local_state            | 4
>>       |
>>   | wsrep_local_state_comment    | Synced
>>      |
>>   | wsrep_cert_index_size        | 3
>>       |
>>   | wsrep_causal_reads           | 0
>>       |
>>   | wsrep_cert_interval          | 0.000000
>>      |
>>   | wsrep_incoming_addresses     | 10.0.0.25:3306,10.0.0.24:3306,
>> 10.0.0.26:3306 |
>>   | wsrep_evs_repl_latency       | 0/0/0/0/0
>>       |
>>   | wsrep_cluster_conf_id        | 21
>>      |
>>   | wsrep_cluster_size           | 3
>>       |
>>   | wsrep_cluster_state_uuid     | f7d5ff68-4482-11e4-9f6c-5f76e5087b34
>>      |
>>   | wsrep_cluster_status         | Primary
>>       |
>>   | wsrep_connected              | ON
>>      |
>>   | wsrep_local_bf_aborts        | 0
>>       |
>>   | wsrep_local_index            | 0
>>       |
>>   | wsrep_provider_name          | Galera
>>      |
>>   | wsrep_provider_vendor        | Codership Oy <email address hidden>
>>       |
>>   | wsrep_provider_version       | 3.7(r7f44a18)
>>       |
>>   | wsrep_ready                  | ON
>>      |
>>
>> +------------------------------+----------------------------------------------+
>>   52 rows in set (0.00 sec)
>>
>>   **** No traces of xtrabackup on the Donor:
>>   ubuntu@ip-10-0-0-25:~$ ps -ef |grep inno
>>   ubuntu   18975 18945  0 07:59 pts/0    00:00:00 grep --color=auto inno
>>   ubuntu@ip-10-0-0-25:~$ ps -ef |grep xtra
>>   ubuntu   18977 18945  0 07:59 pts/0    00:00:00 grep --color=auto xtra
>>   ubuntu@ip-10-0-0-25:~$ ps -ef |grep mysqld
>>   root      9682     1  0 07:10 ?        00:00:00 /bin/sh
>> /usr/bin/mysqld_safe
>>   mysql    10525  9682  0 07:10 ?        00:00:04 /usr/sbin/mysqld
>> --basedir=/usr/ --datadir=/mnt/data/mysql/
>> --plugin-dir=/usr//lib/mysql/plugin --user=mysql
>> --wsrep-provider=/usr/lib/libgalera_smm.so --log-error=/var/log/mysql.log
>> --pid-file=/mnt/data/mysql//mysqld.pid --socket=/var/run/mysqld/mysqld.sock
>> --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
>>
>>
>>   *** innobackup.backup.log on the Donor:
>>
>>   ubuntu@ip-10-0-0-25:~$ sudo cat /mnt/data/mysql/innobackup.backup.log
>>   sudo: unable to resolve host ip-10-0-0-25
>>
>>   InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
>>   and Percona LLC and/or its affiliates 2009-2013.  All Rights Reserved.
>>
>>   This software is published under
>>   the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
>>
>>   Get the latest version of Percona XtraBackup, documentation, and help
>> resources:
>>   http://www.percona.com/xb/p
>>
>>   140925 07:54:09  innobackupex: Connecting to MySQL server with DSN
>> 'dbi:mysql:;mysql_read_default_file=/etc/mysql/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/run/mysqld/mysqld.sock'
>> as 'root'  (using password: YES).
>>   140925 07:54:09  innobackupex: Connected to MySQL server
>>   IMPORTANT: Please check that the backup run completes successfully.
>>              At the end of a successful backup run innobackupex
>>              prints "completed OK!".
>>
>>   innobackupex: Using mysql server version 5.6.20-68.0-56
>>
>>   innobackupex: Created backup directory /tmp/tmp.o9h80diUhb
>>
>>   140925 07:54:09  innobackupex: Starting ibbackup with command:
>> xtrabackup  --defaults-file="/etc/mysql/my.cnf"  --defaults-group="mysqld"
>> --backup --suspend-at-end --target-dir=/tmp/tmp.nnC6BnypNz
>> --tmpdir=/tmp/tmp.nnC6BnypNz --extra-lsndir='/tmp/tmp.nnC6BnypNz'
>> --stream=xbstream
>>   innobackupex: Waiting for ibbackup (pid=18437) to suspend
>>   innobackupex: Suspend file '/tmp/tmp.nnC6BnypNz/xtrabackup_suspended_2'
>>
>>   xtrabackup version 2.2.4 based on MySQL server 5.6.17 Linux (x86_64)
>> (revision id: )
>>   xtrabackup: uses posix_fadvise().
>>   xtrabackup: cd to /mnt/data/mysql/
>>   xtrabackup: open files limit requested 0, set to 5000
>>   xtrabackup: using the following InnoDB configuration:
>>   xtrabackup:   innodb_data_home_dir = ./
>>   xtrabackup:   innodb_data_file_path = ibdata1:100M:autoextend
>>   xtrabackup:   innodb_log_group_home_dir = ./
>>   xtrabackup:   innodb_log_files_in_group = 2
>>   xtrabackup:   innodb_log_file_size = 536870912
>>   xtrabackup: using O_DIRECT
>>   >> log scanned up to (1627118)
>>   [01] Streaming ./ibdata1
>>   [01]        ...done
>>   [01] Streaming ./mysql/innodb_index_stats.ibd
>>   [01]        ...done
>>   [01] Streaming ./mysql/innodb_table_stats.ibd
>>   [01]        ...done
>>   [01] Streaming ./mysql/slave_master_info.ibd
>>   [01]        ...done
>>   [01] Streaming ./mysql/slave_relay_log_info.ibd
>>   [01]        ...done
>>   [01] Streaming ./mysql/slave_worker_info.ibd
>>   [01]        ...done
>>   >> log scanned up to (1627118)
>>   xtrabackup: Creating suspend file
>> '/tmp/tmp.nnC6BnypNz/xtrabackup_suspended_2' with pid '18438'
>>
>>   140925 07:54:11  innobackupex: Continuing after ibbackup has suspended
>>   140925 07:54:11  innobackupex: Executing FLUSH TABLES WITH READ LOCK...
>>   140925 07:54:11  innobackupex: All tables locked and flushed to disk
>>
>>   140925 07:54:11  innobackupex: Starting to backup non-InnoDB tables and
>> files
>>   innobackupex: in subdirectories of '/mnt/data/mysql/'
>>   innobackupex: Backing up files
>> '/mnt/data/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}'
>> (53 files)
>>   >> log scanned up to (1627118)
>>   innobackupex: Backing up files
>> '/mnt/data/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}'
>> (74 files)
>>   140925 07:54:11  innobackupex: Finished backing up non-InnoDB tables and
>> files
>>
>>   140925 07:54:11  innobackupex: Executing FLUSH ENGINE LOGS...
>>   140925 07:54:11  innobackupex: Waiting for log copying to finish
>>
>>   xtrabackup: The latest check point (for incremental): '1627118'
>>   xtrabackup: Stopping log copying thread.
>>   .>> log scanned up to (1627118)
>>
>>   xtrabackup: Creating suspend file
>> '/tmp/tmp.nnC6BnypNz/xtrabackup_log_copied' with pid '18438'
>>   xtrabackup: Transaction log of lsn (1627118) to (1627118) was copied.
>>   140925 07:54:12  innobackupex: All tables unlocked
>>
>>   innobackupex: Backup created in directory '/tmp/tmp.o9h80diUhb'
>>   140925 07:54:12  innobackupex: Connection to database server closed
>>   140925 07:54:12  innobackupex: completed OK!
>>   ubuntu@ip-10-0-0-25:~$
>>
>>
>>   *********** Workaround: *************
>>   - downgrade all nodes to percona xtrabackup 2.2.3, the additional
>> message written in the mysql error log is the following when using
>> xtrabackup 2.2.3:
>>   2014-09-25 08:02:06 30369 [Note] WSREP: Synchronized with group, ready
>> for connections
>>
>>   - use wsrep_sst_method=rsync
>>
>>
>>   ****** If it helps i can facilitate remote login (ssh), it is just
>> simple test servers.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/codership-mysql/+bug/1373796/+subscriptions
>>


-- 
David Bennett, QA Engineer, Percona
Timezone: CST/CDT (UTC-6/UTC-5)
Tel: +1 (208) 473-2904 x654
Mob: +1 (816) 582-5317
Skype: percona.dbennett
PGP/GnuPG: FFD182BB
http://www.percona.com/services.html

Looking for Replication with Data Consistency?
Try Percona XtraDB Cluster
<http://www.percona.com/software/percona-xtradb-cluster>!