May 12 04:07:52 prod-xdb-ams-a03 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql May 12 04:07:52 prod-xdb-ams-a03 mysqld_safe: Skipping wsrep-recover for 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 pair May 12 04:07:52 prod-xdb-ams-a03 mysqld_safe: Assigning 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 to wsrep_start_position May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 0 [Note] WSREP: wsrep_start_position var submitted: '5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362' May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Read nil XID from storage engines, skipping position init May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: wsrep_load(): Galera 3.9(r93aca2d) by Codership Oy loaded successfully. May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: CRC-32C: using hardware acceleration. May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Found saved state: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 10.60.0.52; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = May 12 04:07:53 prod-xdb-ams-a03 mysqld: false; pc.ignore_sb = false; p May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Service thread queue flushed. May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Assign initial position for certification: 143604362, protocol version: -1 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: wsrep_sst_grab() May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Start replication May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Setting initial position to 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: protonet asio version 0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Using CRC-32C for message checksums. May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: backend: asio May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory) May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: restore pc from disk failed May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: GMCast version 0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: EVS version 0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: gcomm: connecting to group 'ams_cluster', peer '10.60.0.50:,10.60.0.51:,10.60.0.52:' May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Warning] WSREP: (764915b6, 'tcp://0.0.0.0:4567') address 'tcp://10.60.0.52:4567' points to own listening address, blacklisting May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') address 'tcp://10.60.0.52:4567' pointing to uuid 764915b6 is blacklisted, skipping May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') address 'tcp://10.60.0.52:4567' pointing to uuid 764915b6 is blacklisted, skipping May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') address 'tcp://10.60.0.52:4567' pointing to uuid 764915b6 is blacklisted, skipping May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: declaring 53e61731 at tcp://10.60.0.50:4567 stable May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: declaring ea8a97de at tcp://10.60.0.51:4567 stable May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: Node 53e61731 state prim May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: view(view_id(PRIM,53e61731,183) memb { May 12 04:07:53 prod-xdb-ams-a03 mysqld: #01153e61731,0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: #011764915b6,0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: #011ea8a97de,0 May 12 04:07:53 prod-xdb-ams-a03 mysqld: } joined { May 12 04:07:53 prod-xdb-ams-a03 mysqld: } left { May 12 04:07:53 prod-xdb-ams-a03 mysqld: } partitioned { May 12 04:07:53 prod-xdb-ams-a03 mysqld: }) May 12 04:07:53 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:53 679 [Note] WSREP: save pc into disk May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: gcomm: connected May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0) May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Opened channel 'ams_cluster' May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3 May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Waiting for SST to complete. May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: STATE EXCHANGE: sent state msg: 76965fd9-f85c-11e4-9040-e36a48889f0b May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: STATE EXCHANGE: got state msg: 76965fd9-f85c-11e4-9040-e36a48889f0b from 0 (prod-xdb-ams-a01) May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: STATE EXCHANGE: got state msg: 76965fd9-f85c-11e4-9040-e36a48889f0b from 2 (prod-xdb-ams-a02) May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: STATE EXCHANGE: got state msg: 76965fd9-f85c-11e4-9040-e36a48889f0b from 1 (prod-xdb-ams-a03) May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Quorum results: May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011version = 3, May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011component = PRIMARY, May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011conf_id = 182, May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011members = 2/3 (joined/total), May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011act_id = 143604412, May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011last_appl. = -1, May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011protocols = 0/7/3 (gcs/repl/appl), May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011group UUID = 5f7e6e38-1ee4-11e4-860a-b38558c9beeb May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Flow-control interval: [28, 28] May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 143604412) May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: State transfer required: May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011Group state: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604412 May 12 04:07:54 prod-xdb-ams-a03 mysqld: #011Local state: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: New cluster view: global state: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604412, view# 183: Primary, number of nodes: 3, my index: 1, protocol version 3 May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Warning] WSREP: Gap in state sequence. Need state transfer. May 12 04:07:54 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:54 679 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.60.0.52' --auth 'sstuser:XXXX' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '679' '' ' May 12 04:07:54 prod-xdb-ams-a03 mysqld: WSREP_SST: [INFO] Logging all stderr of SST/Innobackupex to syslog (20150512 04:07:54.939) ==> /var/log/syslog <== May 12 04:07:54 prod-xdb-ams-a03 -wsrep-sst-joiner: Streaming with xbstream May 12 04:07:54 prod-xdb-ams-a03 -wsrep-sst-joiner: Using socat as streamer May 12 04:07:54 prod-xdb-ams-a03 -wsrep-sst-joiner: Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) ==> /var/log/mysql.log <== May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.60.0.52:4444/xtrabackup_sst//1 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: REPL Protocols: 7 (3, 2) May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Service thread queue flushed. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Assign initial position for certification: 143604412, protocol version: 3 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Service thread queue flushed. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.60.0.52:4568 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Member 1.0 (prod-xdb-ams-a03) requested state transfer from '*any*'. Selected 2.0 (prod-xdb-ams-a02)(SYNCED) as donor. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 143604422) May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Requesting state transfer: success, donor: 2 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: 2.0 (prod-xdb-ams-a02): State transfer to 1.0 (prod-xdb-ams-a03) complete. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: Member 2.0 (prod-xdb-ams-a02) synced with group. ==> /var/log/syslog <== May 12 04:07:55 prod-xdb-ams-a03 -wsrep-sst-joiner: Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) May 12 04:07:55 prod-xdb-ams-a03 -wsrep-sst-joiner: xtrabackup_ist received from donor: Running IST May 12 04:07:55 prod-xdb-ams-a03 -wsrep-sst-joiner: Galera co-ords from recovery: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 May 12 04:07:55 prod-xdb-ams-a03 -wsrep-sst-joiner: Total time on joiner: 0 seconds May 12 04:07:55 prod-xdb-ams-a03 -wsrep-sst-joiner: Removing the sst_in_progress file ==> /var/log/mysql.log <== May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] WSREP: SST complete, seqno: 143604362 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] Plugin 'FEDERATED' is disabled. May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Using atomics to ref count buffer pool pages May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: The InnoDB memory heap is disabled May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Memory barrier is not used May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Compressed tables use zlib 1.2.7 May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Using Linux native AIO May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Using CPU crc32 instructions May 12 04:07:55 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:55 679 [Note] InnoDB: Initializing buffer pool, size = 4.0G May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] InnoDB: Completed initialization of buffer pool May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] InnoDB: Highest supported file format is Barracuda. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] InnoDB: 128 rollback segment(s) are active. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] InnoDB: Waiting for purge to start May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 186453382178 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] Server hostname (bind-address): '*'; port: 3306 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] IPv6 is available. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] - '::' resolves to '::'; May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] Server socket created on IP: '::'. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] Event Scheduler: Loaded 0 events May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Signalling provider to continue. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: inited wsrep sidno 1 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: SST received: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604362 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] /usr/sbin/mysqld: ready for connections. May 12 04:07:56 prod-xdb-ams-a03 mysqld: Version: '5.6.22-72.0-56-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel72.0, Revision 978, WSREP version 25.8, wsrep_25.8.r4150 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Receiving IST: 50 writesets, seqnos 143604362-143604412 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: IST received: 5f7e6e38-1ee4-11e4-860a-b38558c9beeb:143604412 May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: 1.0 (prod-xdb-ams-a03): State transfer from 2.0 (prod-xdb-ams-a02) complete. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Shifting JOINER -> JOINED (TO: 143604431) May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Member 1.0 (prod-xdb-ams-a03) synced with group. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 143604431) May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: Synchronized with group, ready for connections May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. May 12 04:07:56 prod-xdb-ams-a03 mysqld: 2015-05-12 04:07:56 679 [Note] WSREP: (764915b6, 'tcp://0.0.0.0:4567') turning message relay requesting off ==> /var/log/syslog <== May 12 04:08:01 prod-xdb-ams-a03 /USR/SBIN/CRON[1209]: (root) CMD (/usr/bin/pgrep mysqld >/dev/null && mysql -e "show full processlist" | grep -v Sleep > /var/log/mysql/processlist/processlist-`date +%H:%M`)