Bug #1208493 “settting wsrep_provider=none hangs” : Bugs : Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

Seppo Jaakola (seppo-jaakola) on 2013-08-05

Changed in codership-mysql:
status:	New → In Progress
importance:	Undecided → Low
assignee:	nobody → Seppo Jaakola (seppo-jaakola)
milestone:	none → 5.5.32-23.7.6

Revision history for this message

Seppo Jaakola (seppo-jaakola) wrote on 2013-08-05:

#1

Fix for MariaDB branch pushed in revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3410

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-08-05:

#2

Is it because the mariadb code has added logic in thd kill?

Revision history for this message

Alex Yurchenko (ayurchen) wrote on 2013-08-08:

#3

Seppo, isn't it a duplicate of https://bugs.launchpad.net/codership-mysql/+bug/1136571 ? I have encountered that at least once. Maybe this fix is also due for MySQL-wsrep?

Revision history for this message

Seppo Jaakola (seppo-jaakola) wrote on 2013-08-09:

#4

Fix, that solves replication stop problem in MariaDB: http://bazaar.launchpad.net/~codership/codership-mysql/5.5-23/revision/3897

Revision history for this message

Seppo Jaakola (seppo-jaakola) wrote on 2013-08-09:

#5

Fix, that solves hanging when killed client connection tries to access LOCK_global_system_variables when replication stopping is progressing: http://bazaar.launchpad.net/~codership/codership-mysql/5.5-23/revision/3898

This resource conflict was hurting both MySQL and MariaDB versions. Also hanging in lp:1136571 seems to be due to this deadlock.

Note that this fix introduces a potential race condition, which will surface if there are several connections issuing wsrep_provider=none.

Raghavendra D Prabhu (raghavendra-prabhu) on 2013-08-09

Changed in percona-xtradb-cluster:
milestone:	none → 5.5.33-23.7.6

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-08-16:

#6

Download full text (39.6 KiB)

It still seems to hang when

'set global wsrep_provider=none' is done.

=================================================
sudo /pxc/bin/mysqld --defaults-file=/pxc/etc/my.cnf.local --basedir=/pxc --user=mysql --wsrep-cluster-address="gcomm://?pc.ignore_sb=true" --wsrep-start-position='dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
[sudo] password for raghavendra:
Sorry, try again.
[sudo] password for raghavendra:
130816 6:44:07 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:4001' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
130816 6:44:07 [Note] WSREP: wsrep_start_position var submitted: 'dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816 6:44:07 [Note] WSREP: Read nil XID from storage engines, skipping position init
130816 6:44:07 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816 6:44:07 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <email address hidden> loaded succesfully.
130816 6:44:07 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816 6:44:07 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816 6:44:07 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816 6:44:07 [Note] WSREP: Assign initial position for certification: 51367, protocol version: -1
130816 6:44:07 [Note] WSREP: wsrep_sst_grab()
130816 6:44:07 [Note] WSREP: Start replication
130816 6:44:07 [Note] WSREP: Setting initial position to dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816 6:44:07 [Note] WSREP: protonet asio version 0
130816 6:44:07 [Note] WSREP: backend: asio
130816 6:44:07 [Note] WSREP: GMCast version 0
130816 6:44:07 [Note] WSREP: (260c3139-0611-11e3-9874-ebebb5b53a4b, 'tcp://127.0.0.1:4010') listening at tcp://127.0.0.1:4010
130816 6:44:07 [Note] WSREP: (260c3139-0611-11e3-9874-ebebb5b53a4b, 'tcp://127.0.0.1:4010') multicast: , ttl: 1
130816 6:44:07 [Note] WSREP: EVS version 0
130816 6:44:07 [Note] WSREP: PC version 0
130816 6:44:07 [Note] WSREP: gcomm: connecting to group 'Archie', peer ''
130816 6:44:07 [Note] WSREP: Node 260c3139-0611-11e3-9874-ebebb5b53a4b state prim
130816 6:44:07 [Note] WSREP: view(view_id(PRIM,260c3139-0611-11e3-9874-ebebb5b53a4b,1) memb {
260c3139-0611-11e3-9874-ebebb5b53a4b,
} joined {
} left {
} partitioned {
})
130816 6:44:07 [Note] WSREP: gcomm: connected
130816 6:44:07 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130816 6:44:07 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130816 6:44:07 [Note] WSRE...

It still seems to hang when

'set global wsrep_provider=none' is done.

=================================================
sudo /pxc/bin/mysqld --defaults-file=/pxc/etc/my.cnf.local --basedir=/pxc --user=mysql --wsrep-cluster-address="gcomm://?pc.ignore_sb=true" --wsrep-start-position='dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
[sudo] password for raghavendra:
Sorry, try again.
[sudo] password for raghavendra:
130816  6:44:07 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:4001' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
130816  6:44:07 [Note] WSREP: wsrep_start_position var submitted: 'dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  6:44:07 [Note] WSREP: Read nil XID from storage engines, skipping position init
130816  6:44:07 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  6:44:07 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  6:44:07 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816  6:44:07 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  6:44:07 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816  6:44:07 [Note] WSREP: Assign initial position for certification: 51367, protocol version: -1
130816  6:44:07 [Note] WSREP: wsrep_sst_grab()
130816  6:44:07 [Note] WSREP: Start replication
130816  6:44:07 [Note] WSREP: Setting initial position to dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816  6:44:07 [Note] WSREP: protonet asio version 0
130816  6:44:07 [Note] WSREP: backend: asio
130816  6:44:07 [Note] WSREP: GMCast version 0
130816  6:44:07 [Note] WSREP: (260c3139-0611-11e3-9874-ebebb5b53a4b, 'tcp://127.0.0.1:4010') listening at tcp://127.0.0.1:4010
130816  6:44:07 [Note] WSREP: (260c3139-0611-11e3-9874-ebebb5b53a4b, 'tcp://127.0.0.1:4010') multicast: , ttl: 1
130816  6:44:07 [Note] WSREP: EVS version 0
130816  6:44:07 [Note] WSREP: PC version 0
130816  6:44:07 [Note] WSREP: gcomm: connecting to group 'Archie', peer ''
130816  6:44:07 [Note] WSREP: Node 260c3139-0611-11e3-9874-ebebb5b53a4b state prim
130816  6:44:07 [Note] WSREP: view(view_id(PRIM,260c3139-0611-11e3-9874-ebebb5b53a4b,1) memb {
        260c3139-0611-11e3-9874-ebebb5b53a4b,
} joined {
} left {
} partitioned {
})
130816  6:44:07 [Note] WSREP: gcomm: connected
130816  6:44:07 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130816  6:44:07 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130816  6:44:07 [Note] WSREP: Opened channel 'Archie'
130816  6:44:07 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130816  6:44:07 [Note] WSREP: Waiting for SST to complete.
130816  6:44:07 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 260ccb83-0611-11e3-852f-3e4cca916918
130816  6:44:07 [Note] WSREP: STATE EXCHANGE: sent state msg: 260ccb83-0611-11e3-852f-3e4cca916918
130816  6:44:07 [Note] WSREP: STATE EXCHANGE: got state msg: 260ccb83-0611-11e3-852f-3e4cca916918 from 0 (Arch1)
130816  6:44:07 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (joined/total),
        act_id     = 51367,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = dd00df13-e87a-11e2-0800-5cbe45ea7ee5
130816  6:44:07 [Note] WSREP: Flow-control interval: [16, 16]
130816  6:44:07 [Note] WSREP: Restored state OPEN -> JOINED (51367)
130816  6:44:07 [Note] WSREP: Member 0 (Arch1) synced with group.
130816  6:44:07 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 51367)
130816  6:44:07 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
130816  6:44:07 [Note] WSREP: SST complete, seqno: 51367
130816  6:44:07 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
130816  6:44:07 InnoDB: The InnoDB memory heap is disabled
130816  6:44:07 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130816  6:44:07 InnoDB: Compressed tables use zlib 1.2.8
130816  6:44:07 InnoDB: Using Linux native AIO
130816  6:44:07 InnoDB: Initializing buffer pool, size = 500.0M
130816  6:44:07 InnoDB: Completed initialization of buffer pool
130816  6:44:07 InnoDB: highest supported file format is Barracuda.
130816  6:44:08  InnoDB: Waiting for the background threads to start
130816  6:44:09 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 107815182
130816  6:44:09 [Note] Event Scheduler: Loaded 0 events
130816  6:44:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:44:09 [Note] /pxc/bin/mysqld: ready for connections.
Version: '5.5.33-23.7.5-debug'  socket: '/pxc/datadir/pxc.sock'  port: 4000  Percona XtraDB Cluster (GPL) 5.5.32-23.7.5, Revision 463, wsrep_23.7.5.r463
130816  6:44:09 [Note] WSREP: Assign initial position for certification: 51367, protocol version: 2
130816  6:44:09 [Note] WSREP: Synchronized with group, ready for connections
130816  6:44:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:47:06 [Note] WSREP: Stop replication
130816  6:47:06 [Note] WSREP: Closing send monitor...
130816  6:47:06 [Note] WSREP: Closed send monitor.
130816  6:47:06 [Note] WSREP: gcomm: terminating thread
130816  6:47:06 [Note] WSREP: gcomm: joining thread
130816  6:47:06 [Note] WSREP: gcomm: closing backend
130816  6:47:06 [Note] WSREP: view((empty))
130816  6:47:06 [Note] WSREP: Received self-leave message.
130816  6:47:06 [Note] WSREP: gcomm: closed
130816  6:47:06 [Note] WSREP: Flow-control interval: [0, 0]
130816  6:47:06 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130816  6:47:06 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 51367)
130816  6:47:06 [Note] WSREP: RECV thread exiting 0: Success
130816  6:47:06 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130816  6:47:06 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:47:06 [Note] WSREP: recv_thread() joined.
130816  6:47:06 [Note] WSREP: Closing replication queue.
130816  6:47:06 [Note] WSREP: Closing slave action queue.
130816  6:47:06 [Note] WSREP: applier thread exiting (code:0)
130816  6:47:06 [Note] WSREP: applier thread exiting (code:5)
130816  6:47:08 [Note] WSREP: rollbacker thread exiting
130816  6:47:08 [Note] WSREP: dtor state: CLOSED
130816  6:47:08 [Note] WSREP: apply mon: entered 0
130816  6:47:08 [Note] WSREP: apply mon: entered 0
130816  6:47:08 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
130816  6:47:08 [Note] WSREP: cert index usage at exit 0
130816  6:47:08 [Note] WSREP: cert trx map usage at exit 0
130816  6:47:08 [Note] WSREP: deps set usage at exit 0
130816  6:47:08 [Note] WSREP: avg deps dist 0
130816  6:47:08 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130816  6:47:08 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 51367)
130816  6:47:08 [Note] WSREP: Flushing memory map to disk...
130816  6:47:08 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816  6:47:08 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  6:47:08 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  6:47:08 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:51367
130816  6:47:08 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  6:47:08 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4010; cert.log_conflicts = no; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT15S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0; evs.view_forget_timeout = PT5M; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; gmcast.mcast_addr = ; gmcast.mcast_
130816  6:47:08 [Note] WSREP: Assign initial position for certification: 51367, protocol version: -1

^\130816  6:49:10 [Note] /pxc/bin/mysqld: Normal shutdown

130816  6:49:10 [Note] WSREP: Stop replication
130816  6:49:12 [Note] Event Scheduler: Purging the queue. 0 events
130816  6:49:14 [Warning] /pxc/bin/mysqld: Forcing close of thread 5  user: ''

^\01:20:50 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=10002
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21957244 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/pxc/bin/mysqld(my_print_stacktrace+0x2c)[0x81148a]
/pxc/bin/mysqld(handle_fatal_signal+0x30f)[0x6d12db]
/usr/lib/libpthread.so.0(+0xf830)[0x7f684c920830]
/usr/lib/libpthread.so.0(pthread_cond_wait+0xbf)[0x7f684c91cfff]
/pxc/bin/mysqld(_Z11mysqld_mainiPPc+0x380e)[0x521bb3]
/pxc/bin/mysqld(main+0x9)[0x515279]
/usr/lib/libc.so.6(__libc_start_main+0xf5)[0x7f684b0b9bc5]
/pxc/bin/mysqld[0x5151a9]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
                                                                                                                                                                                                 (/tmp)~6:50-[1]
>>sudo /pxc/bin/mysqld --defaults-file=/pxc/etc/my.cnf.local --basedir=/pxc --user=mysql --wsrep-cluster-address="gcomm://?pc.ignore_sb=true" --wsrep-start-position='dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  6:51:01 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:4001' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
130816  6:51:01 [Note] WSREP: wsrep_start_position var submitted: 'dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  6:51:01 [Note] WSREP: Read nil XID from storage engines, skipping position init
130816  6:51:01 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  6:51:01 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  6:51:01 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:-1
130816  6:51:01 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  6:51:01 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816  6:51:01 [Note] WSREP: Assign initial position for certification: 38570, protocol version: -1
130816  6:51:01 [Note] WSREP: wsrep_sst_grab()
130816  6:51:01 [Note] WSREP: Start replication
130816  6:51:01 [Note] WSREP: Setting initial position to dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:51:01 [Note] WSREP: protonet asio version 0
130816  6:51:01 [Note] WSREP: backend: asio
130816  6:51:01 [Note] WSREP: GMCast version 0
130816  6:51:01 [Note] WSREP: (1d1b4a29-0612-11e3-889d-87c294671e75, 'tcp://127.0.0.1:4010') listening at tcp://127.0.0.1:4010
130816  6:51:01 [Note] WSREP: (1d1b4a29-0612-11e3-889d-87c294671e75, 'tcp://127.0.0.1:4010') multicast: , ttl: 1
130816  6:51:01 [Note] WSREP: EVS version 0
130816  6:51:01 [Note] WSREP: PC version 0
130816  6:51:01 [Note] WSREP: gcomm: connecting to group 'Archie', peer ''
130816  6:51:01 [Note] WSREP: Node 1d1b4a29-0612-11e3-889d-87c294671e75 state prim
130816  6:51:01 [Note] WSREP: view(view_id(PRIM,1d1b4a29-0612-11e3-889d-87c294671e75,1) memb {
        1d1b4a29-0612-11e3-889d-87c294671e75,
} joined {
} left {
} partitioned {
})
130816  6:51:01 [Note] WSREP: gcomm: connected
130816  6:51:01 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130816  6:51:01 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130816  6:51:01 [Note] WSREP: Opened channel 'Archie'
130816  6:51:01 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130816  6:51:01 [Note] WSREP: Waiting for SST to complete.
130816  6:51:01 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1d1bd3e7-0612-11e3-9ce7-e33d604d789a
130816  6:51:01 [Note] WSREP: STATE EXCHANGE: sent state msg: 1d1bd3e7-0612-11e3-9ce7-e33d604d789a
130816  6:51:01 [Note] WSREP: STATE EXCHANGE: got state msg: 1d1bd3e7-0612-11e3-9ce7-e33d604d789a from 0 (Arch1)
130816  6:51:01 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (joined/total),
        act_id     = 38570,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = dd00df13-e87a-11e2-0800-5cbe45ea7ee5
130816  6:51:01 [Note] WSREP: Flow-control interval: [16, 16]
130816  6:51:01 [Note] WSREP: Restored state OPEN -> JOINED (38570)
130816  6:51:01 [Note] WSREP: Member 0 (Arch1) synced with group.
130816  6:51:01 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 38570)
130816  6:51:01 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
130816  6:51:01 [Note] WSREP: SST complete, seqno: 38570
130816  6:51:01 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
130816  6:51:01 InnoDB: The InnoDB memory heap is disabled
130816  6:51:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130816  6:51:01 InnoDB: Compressed tables use zlib 1.2.8
130816  6:51:01 InnoDB: Using Linux native AIO
130816  6:51:01 InnoDB: Initializing buffer pool, size = 500.0M
130816  6:51:01 InnoDB: Completed initialization of buffer pool
130816  6:51:01 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 107815182
130816  6:51:01  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 107815336
130816  6:51:02  InnoDB: Waiting for the background threads to start
130816  6:51:03 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 107815336
130816  6:51:03 [Note] Event Scheduler: Loaded 0 events
130816  6:51:03 [Note] /pxc/bin/mysqld: ready for connections.
Version: '5.5.33-23.7.5-debug'  socket: '/pxc/datadir/pxc.sock'  port: 4000  Percona XtraDB Cluster (GPL) 5.5.32-23.7.5, Revision 463, wsrep_23.7.5.r463
130816  6:51:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:51:03 [Note] WSREP: Assign initial position for certification: 38570, protocol version: 2
130816  6:51:03 [Note] WSREP: Synchronized with group, ready for connections
130816  6:51:03 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:51:13 [Note] WSREP: Stop replication
130816  6:51:13 [Note] WSREP: Closing send monitor...
130816  6:51:13 [Note] WSREP: Closed send monitor.
130816  6:51:13 [Note] WSREP: gcomm: terminating thread
130816  6:51:13 [Note] WSREP: gcomm: joining thread
130816  6:51:13 [Note] WSREP: gcomm: closing backend
130816  6:51:13 [Note] WSREP: view((empty))
130816  6:51:13 [Note] WSREP: Received self-leave message.
130816  6:51:13 [Note] WSREP: gcomm: closed
130816  6:51:13 [Note] WSREP: Flow-control interval: [0, 0]
130816  6:51:13 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130816  6:51:13 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 38570)
130816  6:51:13 [Note] WSREP: RECV thread exiting 0: Success
130816  6:51:13 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130816  6:51:13 [Note] WSREP: recv_thread() joined.
130816  6:51:13 [Note] WSREP: Closing replication queue.
130816  6:51:13 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:51:13 [Note] WSREP: Closing slave action queue.
130816  6:51:13 [Note] WSREP: applier thread exiting (code:0)
130816  6:51:13 [Note] WSREP: applier thread exiting (code:5)
130816  6:51:15 [Note] WSREP: rollbacker thread exiting
130816  6:51:15 [Note] WSREP: dtor state: CLOSED
130816  6:51:15 [Note] WSREP: apply mon: entered 0
130816  6:51:15 [Note] WSREP: apply mon: entered 0
130816  6:51:15 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
130816  6:51:15 [Note] WSREP: cert index usage at exit 0
130816  6:51:15 [Note] WSREP: cert trx map usage at exit 0
130816  6:51:15 [Note] WSREP: deps set usage at exit 0
130816  6:51:15 [Note] WSREP: avg deps dist 0
130816  6:51:15 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130816  6:51:15 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 38570)
130816  6:51:15 [Note] WSREP: Flushing memory map to disk...
130816  6:51:15 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:51:15 [Note] WSREP: wsrep_load(): loading provider library 'none'
130816  6:51:15 [ERROR] WSREP: Failed to get provider options
130816  6:51:23 [Note] WSREP: Stop replication
130816  6:51:25 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:51:25 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  6:51:25 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  6:51:25 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:51:25 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  6:51:25 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816  6:51:25 [Note] WSREP: Assign initial position for certification: 38570, protocol version: -1
01:22:30 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=10002
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21957244 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/pxc/bin/mysqld(my_print_stacktrace+0x2c)[0x81148a]
/pxc/bin/mysqld(handle_fatal_signal+0x30f)[0x6d12db]
/usr/lib/libpthread.so.0(+0xf830)[0x7fd72f68c830]
/usr/lib/libc.so.6(__poll+0x2d)[0x7fd72dedffdd]
/pxc/bin/mysqld(_Z26handle_connections_socketsv+0x150)[0x51da02]
/pxc/bin/mysqld(_Z11mysqld_mainiPPc+0x33a8)[0x52174d]
/pxc/bin/mysqld(main+0x9)[0x515279]
/usr/lib/libc.so.6(__libc_start_main+0xf5)[0x7fd72de25bc5]
/pxc/bin/mysqld[0x5151a9]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
                                                                                                                                                                                                 (/tmp)~6:52-[1]
>>sudo /pxc/bin/mysqld --defaults-file=/pxc/etc/my.cnf.local --basedir=/pxc --user=mysql --wsrep-cluster-address="gcomm://?pc.ignore_sb=true" --wsrep-start-position='dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  6:54:12 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:4001' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
130816  6:54:12 [Note] WSREP: wsrep_start_position var submitted: 'dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  6:54:12 [Note] WSREP: Read nil XID from storage engines, skipping position init
130816  6:54:12 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  6:54:12 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  6:54:12 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:-1
130816  6:54:12 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  6:54:12 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816  6:54:12 [Note] WSREP: Assign initial position for certification: 38570, protocol version: -1
130816  6:54:12 [Note] WSREP: wsrep_sst_grab()
130816  6:54:12 [Note] WSREP: Start replication
130816  6:54:12 [Note] WSREP: Setting initial position to dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:54:12 [Note] WSREP: protonet asio version 0
130816  6:54:12 [Note] WSREP: backend: asio
130816  6:54:12 [Note] WSREP: GMCast version 0
130816  6:54:12 [Note] WSREP: (8ea38fdb-0612-11e3-862d-533d8b0bb429, 'tcp://127.0.0.1:4010') listening at tcp://127.0.0.1:4010
130816  6:54:12 [Note] WSREP: (8ea38fdb-0612-11e3-862d-533d8b0bb429, 'tcp://127.0.0.1:4010') multicast: , ttl: 1
130816  6:54:12 [Note] WSREP: EVS version 0
130816  6:54:12 [Note] WSREP: PC version 0
130816  6:54:12 [Note] WSREP: gcomm: connecting to group 'Archie', peer ''
130816  6:54:12 [Note] WSREP: Node 8ea38fdb-0612-11e3-862d-533d8b0bb429 state prim
130816  6:54:12 [Note] WSREP: view(view_id(PRIM,8ea38fdb-0612-11e3-862d-533d8b0bb429,1) memb {
        8ea38fdb-0612-11e3-862d-533d8b0bb429,
} joined {
} left {
} partitioned {
})
130816  6:54:12 [Note] WSREP: gcomm: connected
130816  6:54:12 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130816  6:54:12 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130816  6:54:12 [Note] WSREP: Opened channel 'Archie'
130816  6:54:12 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130816  6:54:12 [Note] WSREP: Waiting for SST to complete.
130816  6:54:12 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 8ea4364c-0612-11e3-a051-163baf217c18
130816  6:54:12 [Note] WSREP: STATE EXCHANGE: sent state msg: 8ea4364c-0612-11e3-a051-163baf217c18
130816  6:54:12 [Note] WSREP: STATE EXCHANGE: got state msg: 8ea4364c-0612-11e3-a051-163baf217c18 from 0 (Arch1)
130816  6:54:12 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (joined/total),
        act_id     = 38570,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = dd00df13-e87a-11e2-0800-5cbe45ea7ee5
130816  6:54:12 [Note] WSREP: Flow-control interval: [16, 16]
130816  6:54:12 [Note] WSREP: Restored state OPEN -> JOINED (38570)
130816  6:54:12 [Note] WSREP: Member 0 (Arch1) synced with group.
130816  6:54:12 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 38570)
130816  6:54:12 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
130816  6:54:12 [Note] WSREP: SST complete, seqno: 38570
130816  6:54:12 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
130816  6:54:12 InnoDB: The InnoDB memory heap is disabled
130816  6:54:12 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130816  6:54:12 InnoDB: Compressed tables use zlib 1.2.8
130816  6:54:12 InnoDB: Using Linux native AIO
130816  6:54:12 InnoDB: Initializing buffer pool, size = 500.0M
130816  6:54:12 InnoDB: Completed initialization of buffer pool
130816  6:54:12 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130816  6:54:12  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
130816  6:54:13  InnoDB: Waiting for the background threads to start
130816  6:54:14 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 107815506
130816  6:54:14 [Note] Event Scheduler: Loaded 0 events
130816  6:54:14 [Note] /pxc/bin/mysqld: ready for connections.
Version: '5.5.33-23.7.5-debug'  socket: '/pxc/datadir/pxc.sock'  port: 4000  Percona XtraDB Cluster (GPL) 5.5.32-23.7.5, Revision 463, wsrep_23.7.5.r463
130816  6:54:14 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:54:14 [Note] WSREP: Assign initial position for certification: 38570, protocol version: 2
130816  6:54:14 [Note] WSREP: Synchronized with group, ready for connections
130816  6:54:14 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:58:10 [Note] WSREP: Stop replication
130816  6:58:10 [Note] WSREP: Closing send monitor...
130816  6:58:10 [Note] WSREP: Closed send monitor.
130816  6:58:10 [Note] WSREP: gcomm: terminating thread
130816  6:58:10 [Note] WSREP: gcomm: joining thread
130816  6:58:10 [Note] WSREP: gcomm: closing backend
130816  6:58:10 [Note] WSREP: view((empty))
130816  6:58:10 [Note] WSREP: Received self-leave message.
130816  6:58:10 [Note] WSREP: gcomm: closed
130816  6:58:10 [Note] WSREP: Flow-control interval: [0, 0]
130816  6:58:10 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130816  6:58:10 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 38570)
130816  6:58:10 [Note] WSREP: RECV thread exiting 0: Success
130816  6:58:10 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130816  6:58:10 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  6:58:10 [Note] WSREP: recv_thread() joined.
130816  6:58:10 [Note] WSREP: applier thread exiting (code:0)
130816  6:58:10 [Note] WSREP: Closing replication queue.
130816  6:58:10 [Note] WSREP: Closing slave action queue.
130816  6:58:10 [Note] WSREP: applier thread exiting (code:5)
130816  6:58:12 [Note] WSREP: rollbacker thread exiting

^\130816  6:59:38 [Note] /pxc/bin/mysqld: Normal shutdown

130816  6:59:38 [Note] WSREP: Stop replication
130816  6:59:38 [Note] WSREP: dtor state: CLOSED
130816  6:59:38 [Note] WSREP: apply mon: entered 0
130816  6:59:38 [Note] WSREP: apply mon: entered 0
130816  6:59:38 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
130816  6:59:38 [Note] WSREP: cert index usage at exit 0
130816  6:59:38 [Note] WSREP: cert trx map usage at exit 0
130816  6:59:38 [Note] WSREP: deps set usage at exit 0
130816  6:59:38 [Note] WSREP: avg deps dist 0
130816  6:59:38 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130816  6:59:38 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 38570)
130816  6:59:38 [Note] WSREP: Flushing memory map to disk...
130816  6:59:38 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  6:59:38 [Note] WSREP: wsrep_load(): loading provider library 'none'
130816  6:59:38 [ERROR] WSREP: Failed to get provider options
130816  6:59:40 [Note] WSREP: killing local connection: 4
130816  6:59:40 [Note] Event Scheduler: Purging the queue. 0 events
130816  6:59:40  InnoDB: Starting shutdown...
130816  6:59:44  InnoDB: Shutdown completed; log sequence number 107815660
130816  6:59:44 [Note] /pxc/bin/mysqld: Shutdown complete

(/tmp)~6:59-0
>>sudo /pxc/bin/mysqld --defaults-file=/pxc/etc/my.cnf.local --basedir=/pxc --user=mysql --wsrep-cluster-address="gcomm://?pc.ignore_sb=true" --wsrep-start-position='dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  7:08:39 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:4001' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
130816  7:08:39 [Note] WSREP: wsrep_start_position var submitted: 'dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570'
130816  7:08:39 [Note] WSREP: Read nil XID from storage engines, skipping position init
130816  7:08:39 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816  7:08:40 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info@codership.com> loaded succesfully.
130816  7:08:40 [Note] WSREP: Found saved state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  7:08:40 [Note] WSREP: Reusing existing '/pxc/datadir//galera.cache'.
130816  7:08:40 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /pxc/datadir/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /pxc/datadir//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:4010; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130816  7:08:40 [Note] WSREP: Assign initial position for certification: 38570, protocol version: -1
130816  7:08:40 [Note] WSREP: wsrep_sst_grab()
130816  7:08:40 [Note] WSREP: Start replication
130816  7:08:40 [Note] WSREP: Setting initial position to dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816  7:08:40 [Note] WSREP: protonet asio version 0
130816  7:08:40 [Note] WSREP: backend: asio
130816  7:08:40 [Note] WSREP: GMCast version 0
130816  7:08:40 [Note] WSREP: (93ce67b7-0614-11e3-ba94-4bfaa265dabd, 'tcp://127.0.0.1:4010') listening at tcp://127.0.0.1:4010
130816  7:08:40 [Note] WSREP: (93ce67b7-0614-11e3-ba94-4bfaa265dabd, 'tcp://127.0.0.1:4010') multicast: , ttl: 1
130816  7:08:40 [Note] WSREP: EVS version 0
130816  7:08:40 [Note] WSREP: PC version 0
130816  7:08:40 [Note] WSREP: gcomm: connecting to group 'Archie', peer ''
130816  7:08:40 [Note] WSREP: Node 93ce67b7-0614-11e3-ba94-4bfaa265dabd state prim
130816  7:08:40 [Note] WSREP: view(view_id(PRIM,93ce67b7-0614-11e3-ba94-4bfaa265dabd,1) memb {
        93ce67b7-0614-11e3-ba94-4bfaa265dabd,
} joined {
} left {
} partitioned {
})
130816  7:08:40 [Note] WSREP: gcomm: connected
130816  7:08:40 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
130816  7:08:40 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
130816  7:08:40 [Note] WSREP: Opened channel 'Archie'
130816  7:08:40 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
130816  7:08:40 [Note] WSREP: Waiting for SST to complete.
130816  7:08:40 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 93cee02e-0614-11e3-a9aa-db835026e0cb
130816  7:08:40 [Note] WSREP: STATE EXCHANGE: sent state msg: 93cee02e-0614-11e3-a9aa-db835026e0cb
130816  7:08:40 [Note] WSREP: STATE EXCHANGE: got state msg: 93cee02e-0614-11e3-a9aa-db835026e0cb from 0 (Arch1)
130816  7:08:40 [Note] WSREP: Quorum results:
        version    = 2,
        component  = PRIMARY,
        conf_id    = 0,
        members    = 1/1 (joined/total),
        act_id     = 38570,
        last_appl. = -1,
        protocols  = 0/4/2 (gcs/repl/appl),
        group UUID = dd00df13-e87a-11e2-0800-5cbe45ea7ee5
130816  7:08:40 [Note] WSREP: Flow-control interval: [16, 16]
130816  7:08:40 [Note] WSREP: Restored state OPEN -> JOINED (38570)
130816  7:08:40 [Note] WSREP: Member 0 (Arch1) synced with group.
130816  7:08:40 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 38570)
130816  7:08:40 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2
130816  7:08:40 [Note] WSREP: SST complete, seqno: 38570
130816  7:08:40 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
130816  7:08:40 InnoDB: The InnoDB memory heap is disabled
130816  7:08:40 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130816  7:08:40 InnoDB: Compressed tables use zlib 1.2.8
130816  7:08:40 InnoDB: Using Linux native AIO
130816  7:08:40 InnoDB: Initializing buffer pool, size = 500.0M
130816  7:08:40 InnoDB: Completed initialization of buffer pool
130816  7:08:44 InnoDB: highest supported file format is Barracuda.
130816  7:08:44  InnoDB: Waiting for the background threads to start
130816  7:08:45 Percona XtraDB (http://www.percona.com) 5.5.33-rel31.0 started; log sequence number 107815660
130816  7:08:45 [Note] Event Scheduler: Loaded 0 events
130816  7:08:45 [Note] /pxc/bin/mysqld: ready for connections.
Version: '5.5.33-23.7.5-debug'  socket: '/pxc/datadir/pxc.sock'  port: 4000  Percona XtraDB Cluster (GPL) 5.5.33-23.7.5, Revision 465, wsrep_23.7.5.r465
130816  7:08:45 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  7:08:45 [Note] WSREP: Assign initial position for certification: 38570, protocol version: 2
130816  7:08:45 [Note] WSREP: Synchronized with group, ready for connections
130816  7:08:45 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  7:08:53 [Note] WSREP: Stop replication
130816  7:08:53 [Note] WSREP: Closing send monitor...
130816  7:08:53 [Note] WSREP: Closed send monitor.
130816  7:08:53 [Note] WSREP: gcomm: terminating thread
130816  7:08:53 [Note] WSREP: gcomm: joining thread
130816  7:08:53 [Note] WSREP: gcomm: closing backend
130816  7:08:53 [Note] WSREP: view((empty))
130816  7:08:53 [Note] WSREP: Received self-leave message.
130816  7:08:53 [Note] WSREP: gcomm: closed
130816  7:08:53 [Note] WSREP: Flow-control interval: [0, 0]
130816  7:08:53 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130816  7:08:53 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 38570)
130816  7:08:53 [Note] WSREP: RECV thread exiting 0: Success
130816  7:08:53 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130816  7:08:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816  7:08:53 [Note] WSREP: recv_thread() joined.
130816  7:08:53 [Note] WSREP: Closing replication queue.
130816  7:08:53 [Note] WSREP: Closing slave action queue.
130816  7:08:53 [Note] WSREP: applier thread exiting (code:0)
130816  7:08:53 [Note] WSREP: applier thread exiting (code:5)
s130816  7:08:55 [Note] WSREP: rollbacker thread exiting
01:53:04 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=4
max_threads=10002
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 21957244 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/pxc/bin/mysqld(my_print_stacktrace+0x2c)[0x81177a]
/pxc/bin/mysqld(handle_fatal_signal+0x30f)[0x6d15cb]
/usr/lib/libpthread.so.0(+0xf830)[0x7f6e068d7830]
/usr/lib/libc.so.6(__poll+0x2d)[0x7f6e0512afdd]
/pxc/bin/mysqld(_Z26handle_connections_socketsv+0x150)[0x51d9b2]
/pxc/bin/mysqld(_Z11mysqld_mainiPPc+0x33a8)[0x5216fd]
/pxc/bin/mysqld(main+0x9)[0x515229]
/usr/lib/libc.so.6(__libc_start_main+0xf5)[0x7f6e05070bc5]
/pxc/bin/mysqld[0x515159]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
   ================================================================================

I manually crashed the hung server with kill -11 with backtrace above.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-08-16:

#7

full-bt Edit (45.8 KiB, text/plain)

Full backtrace from a hung server.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-08-16:

#8

However, sending SIGQUIT explicitly (with Ctrl-\) makes it shutdown, so this is fine I guess.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-08-16:

#9

Download full text (4.5 KiB)

Ignore #6, and #7, they seem to comply with behavior in #8,

however, setting wsrep_provider to galera library doesn't go down well.

=============================
Version: '5.5.33-23.7.5-debug' socket: '/pxc/datadir/pxc.sock' port: 4000 Percona XtraDB Cluster (GPL) 5.5.33-23.7.5, Revision 465, wsrep_23.7.5.r465
130816 7:30:52 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816 7:30:52 [Note] WSREP: Assign initial position for certification: 38570, protocol version: 2
130816 7:30:52 [Note] WSREP: Synchronized with group, ready for connections
130816 7:30:52 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816 7:30:54 [Note] WSREP: Stop replication
130816 7:30:54 [Note] WSREP: Closing send monitor...
130816 7:30:54 [Note] WSREP: Closed send monitor.
130816 7:30:54 [Note] WSREP: gcomm: terminating thread
130816 7:30:54 [Note] WSREP: gcomm: joining thread
130816 7:30:54 [Note] WSREP: gcomm: closing backend
130816 7:30:54 [Note] WSREP: view((empty))
130816 7:30:54 [Note] WSREP: Received self-leave message.
130816 7:30:54 [Note] WSREP: gcomm: closed
130816 7:30:54 [Note] WSREP: Flow-control interval: [0, 0]
130816 7:30:54 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130816 7:30:54 [Note] WSREP: Shifting SYNCED -> CLOSED (TO: 38570)
130816 7:30:54 [Note] WSREP: RECV thread exiting 0: Success
130816 7:30:54 [Note] WSREP: New cluster view: global state: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130816 7:30:54 [Note] WSREP: recv_thread() joined.
130816 7:30:54 [Note] WSREP: Closing replication queue.
130816 7:30:54 [Note] WSREP: Closing slave action queue.
130816 7:30:54 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130816 7:30:54 [Note] WSREP: applier thread exiting (code:0)
130816 7:30:54 [Note] WSREP: applier thread exiting (code:5)
130816 7:30:56 [Note] WSREP: rollbacker thread exiting
130816 7:30:56 [Note] WSREP: dtor state: CLOSED
130816 7:30:56 [Note] WSREP: apply mon: entered 0
130816 7:30:56 [Note] WSREP: apply mon: entered 0
130816 7:30:56 [Note] WSREP: mon: entered 3 oooe fraction 0 oool fraction 0
130816 7:30:56 [Note] WSREP: cert index usage at exit 0
130816 7:30:56 [Note] WSREP: cert trx map usage at exit 0
130816 7:30:56 [Note] WSREP: deps set usage at exit 0
130816 7:30:56 [Note] WSREP: avg deps dist 0
130816 7:30:56 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130816 7:30:56 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 38570)
130816 7:30:56 [Note] WSREP: Flushing memory map to disk...
130816 7:30:56 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816 7:30:56 [Note] WSREP: wsrep_load(): loading provider library 'none'
130816 7:30:56 [ERROR] WSREP: Failed to get provider options
130816 7:31:10 [Note] WSREP: Stop replication
130816 7:31:12 [Note] WSREP: Initial position: dd00df13-e87a-11e2-0800-5cbe45ea7ee5:38570
130816 7:31:12 [Note] WSREP: wsrep_load(): loading provider library '/pxc/lib/libgalera_smm.so'
130816 7:31:12 [Note] WSREP: wsrep_load(): Galera 2.6(r300) by Codership Oy <info...

	Status	Importance	Assigned to	Milestone
MySQL patches by Codership	Fix Committed	Low	Seppo Jaakola
5.5	Fix Committed	Low	Seppo Jaakola	MySQL patches by Codership 5.5.38-25.11
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC	Status tracked in 5.6
5.5	Fix Released	Undecided	Unassigned
5.6	Fix Released	Undecided	Unassigned

Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

settting wsrep_provider=none hangs

Bug Description

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Changed in codership-mysql:
milestone:	5.5.33-23.7.6 → none

Changed in percona-xtradb-cluster:
milestone:	5.5.33-23.7.6 → future-5.5

Changed in codership-mysql:
status:	In Progress → Fix Committed