Here's what I got on the node that was executed:: 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from read_completion_condition:474 socket 0x7f041400d2d8 13 error 1 1 state 2 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint tcp://192.168.70.2:4567 remote endpoint tcp://192.168.70.3:43628 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: OK -> FAILED 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=749ac05c-97d9-11e3-9ae6-0f4dd301e31f,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=1,tp=0x7f041400d230,ts=4406011659568 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4407096155576 for tcp://192.168.70.3:4567 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d2d8 state 3 send_q size 0 2014-02-17 14:15:25 5046 [Note] WSREP: gcomm/src/gmcast.cpp:update_addresses():805: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') address 'tcp://192.168.70.2:4567' pointing to uuid 9d24bc26-97d8-11e3-860f-fab1d79f1e19 is blacklisted, skipping 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=3ef17bf4-97da-11e3-8b92-57d14582fd80,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=OK,pr=1,tp=0x7f0414003160,ts=4406011676004 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 1 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from read_handler:373 socket 0x7f041400d2d8 -1 error 1 0 state 0 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint tcp://192.168.70.2:4567 remote endpoint tcp://192.168.70.3:43628 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d2d8 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from read_completion_condition:474 socket 0x7f0414003208 14 error 1 1 state 2 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint tcp://192.168.70.2:4567 remote endpoint tcp://192.168.70.4:58637 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: OK -> FAILED 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=3ef17bf4-97da-11e3-8b92-57d14582fd80,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=1,tp=0x7f0414003160,ts=4406011676004 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4407127106308 for tcp://192.168.70.4:4567 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f0414003208 state 3 send_q size 0 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from read_handler:373 socket 0x7f0414003208 -1 error 1 0 state 0 2014-02-17 14:15:25 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint tcp://192.168.70.2:4567 remote endpoint tcp://192.168.70.4:58637 2014-02-17 14:15:25 5046 [Note] WSREP: gcomm/src/gmcast.cpp:check_liveness():1085: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.70.3:4567 tcp://192.168.70.4:4567 2014-02-17 14:15:26 5046 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():954: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') reconnecting to 749ac05c-97d9-11e3-9ae6-0f4dd301e31f (tcp://192.168.70.3:4567), attempt 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.3:4567 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d2d8 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:26 5046 [Note] WSREP: gcomm/src/gmcast.cpp:reconnect():954: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') reconnecting to 3ef17bf4-97da-11e3-8b92-57d14582fd80 (tcp://192.168.70.4:4567), attempt 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.4:4567 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f0414001e48 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d2d8 13 error 1 1 state 1 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d230,ts=4407548285233 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4408550867622 for tcp://192.168.70.3:4567 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d2d8 state 3 send_q size 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=HANDSHAKE_WAIT,pr=0,tp=0x7f0414001da0,ts=4407550446006 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d2d8 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f0414001e48 14 error 1 1 state 1 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f0414001da0,ts=4407550446006 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4408551750369 for tcp://192.168.70.4:4567 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f0414001e48 state 3 send_q size 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:26 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f0414001e48 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.3:4567 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d2d8 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.4:4567 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f0414001e48 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d2d8 13 error 1 1 state 1 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d230,ts=4409046391307 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4410047949343 for tcp://192.168.70.3:4567 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d2d8 state 3 send_q size 0 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=HANDSHAKE_WAIT,pr=0,tp=0x7f0414001da0,ts=4409047541227 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d2d8 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f0414001e48 14 error 1 1 state 1 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f0414001da0,ts=4409047541227 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4410049231562 for tcp://192.168.70.4:4567 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f0414001e48 state 3 send_q size 0 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:28 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f0414001e48 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.3:4567 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d2d8 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.4:4567 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d138 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d2d8 13 error 1 1 state 1 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d230,ts=4410546726707 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4411547747408 for tcp://192.168.70.3:4567 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d2d8 state 3 send_q size 0 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=HANDSHAKE_WAIT,pr=0,tp=0x7f041400d090,ts=4410547384795 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d2d8 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d138 14 error 1 1 state 1 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d090,ts=4410547384795 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4411549282136 for tcp://192.168.70.4:4567 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d138 state 3 send_q size 0 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:29 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d138 2014-02-17 14:15:30 5046 [Note] [Debug] WSREP: gcomm/src/evs_node.hpp:operator()():168: declaring node with index 0 suspected, timeout PT5S 2014-02-17 14:15:30 5046 [Note] [Debug] WSREP: gcomm/src/evs_node.hpp:operator()():168: declaring node with index 1 suspected, timeout PT5S 2014-02-17 14:15:30 5046 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():789: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, OPERATIONAL, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)) suspecting node: 3ef17bf4-97da-11e3-8b92-57d14582fd80 2014-02-17 14:15:30 5046 [Note] WSREP: gcomm/src/evs_proto.cpp:check_inactive():789: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, OPERATIONAL, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)) suspecting node: 749ac05c-97d9-11e3-9ae6-0f4dd301e31f 2014-02-17 14:15:30 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, OPERATIONAL, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 3ef17bf4-97da-11e3-8b92-57d14582fd80 inactive 2014-02-17 14:15:30 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, OPERATIONAL, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 749ac05c-97d9-11e3-9ae6-0f4dd301e31f inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_node.hpp:operator()():182: declaring node with index 0 inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_node.hpp:operator()():182: declaring node with index 1 inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, GATHER, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 3ef17bf4-97da-11e3-8b92-57d14582fd80 inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, GATHER, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 749ac05c-97d9-11e3-9ae6-0f4dd301e31f inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, GATHER, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 3ef17bf4-97da-11e3-8b92-57d14582fd80 inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:set_inactive():845: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, GATHER, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): setting 749ac05c-97d9-11e3-9ae6-0f4dd301e31f inactive 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.3:4567 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d2d8 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.4:4567 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d138 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d2d8 13 error 1 1 state 1 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d230,ts=4412047797534 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4413049459217 for tcp://192.168.70.3:4567 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d2d8 state 3 send_q size 0 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=HANDSHAKE_WAIT,pr=0,tp=0x7f041400d090,ts=4412048527525 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d2d8 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d138 14 error 1 1 state 1 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d090,ts=4412048527525 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4413050547844 for tcp://192.168.70.4:4567 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d138 state 3 send_q size 0 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d138 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:handle_gap():2968: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, GATHER, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)) commit gap from 9d24bc26-97d8-11e3-860f-fab1d79f1e19 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_proto.cpp:handle_gap():3000: evs::proto(9d24bc26-97d8-11e3-860f-fab1d79f1e19, INSTALL, view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15)): install gap evs::msg{version=0,type=3,user_type=255,order=1,seq=-1,seq_range=-1,aru_seq=-1,flags=0,source=9d24bc26-97d8-11e3-860f-fab1d79f1e19,source_view_id=view_id(REG,9d24bc26-97d8-11e3-860f-fab1d79f1e19,16),range_uuid=00000000-0000-0000-0000-000000000000,range=[-1,-1],fifo_seq=13202,node_list=() } 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:handle_view():538: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 view(view_id(TRANS,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:handle_trans():451: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 current view view(view_id(REG,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 } left { } partitioned { }) next view view(view_id(TRANS,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) pc view view(view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { }) 2014-02-17 14:15:31 5046 [Note] WSREP: gcomm/src/pc_proto.cpp:deliver_view():224: view(view_id(NON_PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_gcomm.cpp:fill_cmp_msg():553: member: 0 uuid: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_gcomm.cpp:fill_cmp_msg():566: my index 0 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs_group.c:gcs_group_handle_comp_msg():400: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_group.c:gcs_group_handle_comp_msg():414: #281: Saving SYNCED over SYNCED 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs.c:_set_fc_limits():744: Flow-control interval: [16, 16] 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_sm.h:gcs_sm_continue():317: Trying to continue unpaused monitor 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs.c:gcs_handle_act_conf():848: Received NON-PRIMARY. 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs.c:gcs_shift_state():550: Shifting SYNCED -> OPEN (TO: 3) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/monitor.hpp:drain_common():409: draining up to 3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/monitor.hpp:drain_common():409: draining up to 3 2014-02-17 14:15:31 5046 [Note] WSREP: New cluster view: global state: 66ccdaa9-97d4-11e3-8e47-537667525fe6:3, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2 2014-02-17 14:15:31 5046 [Note] WSREP: Setting wsrep_ready to 0 2014-02-17 14:15:31 5046 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/saved_state.cpp:write_and_flush():222: Saving state: 66ccdaa9-97d4-11e3-8e47-537667525fe6:3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1441: GMCast::handle_stable_view: view(view_id(NON_PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:shift_to():321: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 shift_to: PRIM -> TRANS prim 0 last prim view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) to_seq 68 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_input_map2.cpp:clear():227: discarding 8 messages from recovery index 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_input_map2.cpp:reset():153: size 0 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/evs_input_map2.cpp:reset():159: node: {idx=0,range=[0,-1],safe_seq=-1} size 1 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1441: GMCast::handle_stable_view: view(view_id(REG,9d24bc26-97d8-11e3-860f-fab1d79f1e19,16) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:handle_view():538: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 view(view_id(REG,9d24bc26-97d8-11e3-860f-fab1d79f1e19,16) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:shift_to():321: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 shift_to: TRANS -> STATES_EXCH prim 0 last prim view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) to_seq 68 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:send_state():123: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 sending state 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:send_state():140: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 local to seq 68 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:send_state():141: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 sending state: pcmsg{ type=STATE, seq=0, flags= 0, node_map { 3ef17bf4-97da-11e3-8b92-57d14582fd80,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=61,weight=1,segment=0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=61,weight=1,segment=0 9d24bc26-97d8-11e3-860f-fab1d79f1e19,prim=0,un=0,last_seq=2,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=68,weight=1,segment=0 }} 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:handle_state():864: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 handle state from 9d24bc26-97d8-11e3-860f-fab1d79f1e19 pcmsg{ type=STATE, seq=0, flags= 0, node_map { 3ef17bf4-97da-11e3-8b92-57d14582fd80,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=61,weight=1,segment=0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=61,weight=1,segment=0 9d24bc26-97d8-11e3-860f-fab1d79f1e19,prim=0,un=0,last_seq=2,last_prim=view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15),to_seq=68,weight=1,segment=0 }} 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:validate_state_msgs():632: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 joining to view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:validate_state_msgs():632: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 joining to view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:is_prim():750: Non-prim 9d24bc26-97d8-11e3-860f-fab1d79f1e19 from view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) joining prim 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:is_prim():831: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 greatest view id view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:is_prim():843: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 intersection size 1 greatest view size 3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/pc_proto.cpp:shift_to():321: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 shift_to: STATES_EXCH -> NON_PRIM prim 0 last prim view_id(PRIM,3ef17bf4-97da-11e3-8b92-57d14582fd80,15) to_seq 68 2014-02-17 14:15:31 5046 [Note] WSREP: gcomm/src/pc_proto.cpp:deliver_view():224: view(view_id(NON_PRIM,9d24bc26-97d8-11e3-860f-fab1d79f1e19,16) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_gcomm.cpp:fill_cmp_msg():553: member: 0 uuid: 9d24bc26-97d8-11e3-860f-fab1d79f1e19 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_gcomm.cpp:fill_cmp_msg():566: my index 0 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs_group.c:gcs_group_handle_comp_msg():400: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs.c:_set_fc_limits():744: Flow-control interval: [16, 16] 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcs/src/gcs_sm.h:gcs_sm_continue():317: Trying to continue unpaused monitor 2014-02-17 14:15:31 5046 [Note] WSREP: gcs/src/gcs.c:gcs_handle_act_conf():848: Received NON-PRIMARY. 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/monitor.hpp:drain_common():409: draining up to 3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/monitor.hpp:drain_common():409: draining up to 3 2014-02-17 14:15:31 5046 [Note] WSREP: New cluster view: global state: 66ccdaa9-97d4-11e3-8e47-537667525fe6:3, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2 2014-02-17 14:15:31 5046 [Note] WSREP: Setting wsrep_ready to 0 2014-02-17 14:15:31 5046 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: galera/src/saved_state.cpp:write_and_flush():222: Saving state: 66ccdaa9-97d4-11e3-8e47-537667525fe6:3 2014-02-17 14:15:31 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_stable_view():1441: GMCast::handle_stable_view: view(view_id(NON_PRIM,9d24bc26-97d8-11e3-860f-fab1d79f1e19,16) memb { 9d24bc26-97d8-11e3-860f-fab1d79f1e19,0 } joined { } left { } partitioned { 3ef17bf4-97da-11e3-8b92-57d14582fd80,0 749ac05c-97d9-11e3-9ae6-0f4dd301e31f,0 }) 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.3:4567 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f041400d508 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: galerautils/src/gu_uri.cpp:parse():137: URI: tcp://192.168.70.4:4567 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:AsioTcpSocket():28: ctor for 0x7f0414004718 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: INIT -> HANDSHAKE_WAIT 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f041400d508 13 error 1 1 state 1 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.3:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f041400d460,ts=4413054692107 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4414056354674 for tcp://192.168.70.3:4567 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f041400d508 state 3 send_q size 0 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():863: Proto: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=HANDSHAKE_WAIT,pr=0,tp=0x7f0414004670,ts=4413055558200 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f041400d508 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():45: failed handler from connect_handler:140 socket 0x7f0414004718 14 error 1 1 state 1 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:failed_handler():52: local endpoint remote endpoint 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_up():1367: socket in state 3 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast_proto.cpp:set_state():13: State change: HANDSHAKE_WAIT -> FAILED 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():637: handle failed: v=0,lu=9d24bc26-97d8-11e3-860f-fab1d79f1e19,ru=00000000-0000-0000-0000-000000000000,ls=0,rs=0,la=tcp://0.0.0.0:4567,ra=tcp://192.168.70.4:4567,mc=,gn=mycluster,ch=0,st=FAILED,pr=0,tp=0x7f0414004670,ts=4413055558200 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:handle_failed():666: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') setting next reconnect time to 4414056925074 for tcp://192.168.70.4:4567 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:close():240: closing 0x7f0414004718 state 3 send_q size 0 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():846: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree begin --- 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():887: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') self index: 0 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/gmcast.cpp:update_addresses():888: (9d24bc26-97d8-11e3-860f-fab1d79f1e19, 'tcp://0.0.0.0:4567') --- mcast tree end --- 2014-02-17 14:15:32 5046 [Note] [Debug] WSREP: gcomm/src/asio_tcp.cpp:~AsioTcpSocket():33: dtor for 0x7f0414004718 On node2: 2014-02-17 14:15:25 7fc660921700 InnoDB: Assertion failure in thread 140490000439040 in file row0mysql.cc line 1705 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON 2014-02-17 14:15:25 7fc660921700 InnoDB: Assertion failure in thread 140490000439040 in file row0mysql.cc line 1705 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:15:25 UTC - mysqld got signal 6 ; 14:15:25 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster key_buffer_size=8388608 key_buffer_size=8388608 read_buffer_size=131072 read_buffer_size=131072 max_used_connections=6 max_used_connections=6 max_threads=153 max_threads=153 thread_count=6 thread_count=6 connection_count=6 connection_count=6 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69222 K bytes of memory It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69222 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x214f620 Thread pointer: 0x214f620 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7fc660920d98 thread_stack 0x40000 stack_bottom = 7fc660920d98 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x901f25] /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x901f25] /usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x67fc54] /usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x67fc54] /lib64/libpthread.so.0(+0xf500)[0x7fc68f380500] /lib64/libpthread.so.0(+0xf500)[0x7fc68f380500] /lib64/libc.so.6(gsignal+0x35)[0x7fc68d9e18e5] /lib64/libc.so.6(gsignal+0x35)[0x7fc68d9e18e5] /lib64/libc.so.6(abort+0x175)[0x7fc68d9e30c5] /lib64/libc.so.6(abort+0x175)[0x7fc68d9e30c5] /usr/sbin/mysqld[0x9fced3] /usr/sbin/mysqld[0x9fced3] /usr/sbin/mysqld[0x9687cf] /usr/sbin/mysqld[0x9687cf] /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x5c7d4f] /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x5c7d4f] /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEPK14Relay_log_info+0x2c)[0x89570c] /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEPK14Relay_log_info+0x2c)[0x89570c] /usr/sbin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x2f)[0x89347f] /usr/sbin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x2f)[0x89347f] /usr/sbin/mysqld(_ZN14Rows_log_event24do_table_scan_and_updateEPK14Relay_log_info+0x1db)[0x89f7eb] /usr/sbin/mysqld(_ZN14Rows_log_event24do_table_scan_and_updateEPK14Relay_log_info+0x1db)[0x89f7eb] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xd8f)[0x8a52bf] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xd8f)[0x8a52bf] /usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x68)[0x8a6518] /usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x68)[0x8a6518] /usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x5ce)[0x5bcebe] /usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x5ce)[0x5bcebe] /usr/lib64/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x7fc6882f93f1] /usr/lib64/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x7fc6882f93f1] /usr/lib64/libgalera_smm.so(+0x1a79e5)[0x7fc6883319e5] /usr/lib64/libgalera_smm.so(+0x1a79e5)[0x7fc6883319e5] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x283)[0x7fc688332853] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x283)[0x7fc688332853] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7fc688333145] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7fc688333145] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2c9)[0x7fc68830c4a9] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2c9)[0x7fc68830c4a9] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7fc68830c983] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7fc68830c983] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x7fc68832ba33] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x7fc68832ba33] /usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7fc688342943] /usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7fc688342943] /usr/sbin/mysqld[0x5bdc4f] /usr/sbin/mysqld[0x5bdc4f] /usr/sbin/mysqld(start_wsrep_THD+0x480)[0x5ae070] /usr/sbin/mysqld(start_wsrep_THD+0x480)[0x5ae070] /lib64/libpthread.so.0(+0x7851)[0x7fc68f378851] /lib64/libpthread.so.0(+0x7851)[0x7fc68f378851] /lib64/libc.so.6(clone+0x6d)[0x7fc68da9794d] /lib64/libc.so.6(clone+0x6d)[0x7fc68da9794d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): Query (0): is an invalid pointer is an invalid pointer Connection ID (thread ID): 3 Connection ID (thread ID): 3 Status: NOT_KILLED Status: NOT_KILLED You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. 140217 14:15:25 mysqld_safe Number of processes running now: 0 140217 14:15:25 mysqld_safe Number of processes running now: 0 140217 14:15:25 mysqld_safe WSREP: not restarting wsrep node automatically 140217 14:15:25 mysqld_safe WSREP: not restarting wsrep node automatically 140217 14:15:25 mysqld_safe mysqld from pid file /var/lib/mysql/node2.pid ended 140217 14:15:25 mysqld_safe mysqld from pid file /var/lib/mysql/node2.pid ended On Node3: 2014-02-17 14:15:25 7f2f1dde5700 InnoDB: Assertion failure in thread 139840341300992 in file row0mysql.cc line 1705 InnoDB: Failing assertion: node->pcur->rel_pos == BTR_PCUR_ON InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:15:25 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=6 max_threads=153 thread_count=6 connection_count=6 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69222 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f2f08000990 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f2f1dde4d98 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x901f25] /usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x67fc54] /lib64/libpthread.so.0(+0xf500)[0x7f2f4a7da500] /lib64/libc.so.6(gsignal+0x35)[0x7f2f48e3b8e5] /lib64/libc.so.6(abort+0x175)[0x7f2f48e3d0c5] /usr/sbin/mysqld[0x9fced3] /usr/sbin/mysqld[0x9687cf] /usr/sbin/mysqld(_ZN7handler13ha_delete_rowEPKh+0xff)[0x5c7d4f] /usr/sbin/mysqld(_ZN21Delete_rows_log_event11do_exec_rowEPK14Relay_log_info+0x2c)[0x89570c] /usr/sbin/mysqld(_ZN14Rows_log_event12do_apply_rowEPK14Relay_log_info+0x2f)[0x89347f] /usr/sbin/mysqld(_ZN14Rows_log_event24do_table_scan_and_updateEPK14Relay_log_info+0x1db)[0x89f7eb] /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xd8f)[0x8a52bf] /usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x68)[0x8a6518] /usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x5ce)[0x5bcebe] /usr/lib64/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x7f2f2969f3f1] /usr/lib64/libgalera_smm.so(+0x1a79e5)[0x7f2f296d79e5] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x283)[0x7f2f296d8853] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7f2f296d9145] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2c9)[0x7f2f296b24a9] /usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7f2f296b2983] /usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x7f2f296d1a33] /usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7f2f296e8943] /usr/sbin/mysqld[0x5bdc4f] /usr/sbin/mysqld(start_wsrep_THD+0x480)[0x5ae070] /lib64/libpthread.so.0(+0x7851)[0x7f2f4a7d2851] /lib64/libc.so.6(clone+0x6d)[0x7f2f48ef194d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): is an invalid pointer Connection ID (thread ID): 1 Status: NOT_KILLED You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. 140217 14:15:25 mysqld_safe Number of processes running now: 0 140217 14:15:25 mysqld_safe WSREP: not restarting wsrep node automatically 140217 14:15:25 mysqld_safe mysqld from pid file /var/lib/mysql/node3.pid ended