Master server stopped with a message: ===================================== 2014-01-20 16:30:39 7f0c7af3b700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 9135 srv_active, 0 srv_shutdown, 51375 srv_idle srv_master_thread log flush and writes: 60503 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 652345 --Thread 139691519579904 has waited at lock0lock.cc line 4766 for 935.00 seconds the semaphore: Mutex at 0x9c86078 '&lock_sys->mutex', lock var 1 waiters flag 1 OS WAIT ARRAY INFO: signal count 2548472 Mutex spin waits 388600, rounds 2017742, OS waits 47705 RW-shared spins 903106, rounds 10322271, OS waits 147512 RW-excl spins 421647, rounds 16753936, OS waits 452880 Spin rounds per wait: 5.19 mutex, 11.43 RW-shared, 39.73 RW-excl FAIL TO OBTAIN LOCK MUTEX, SKIP LOCK INFO PRINTING -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (read thread) I/O thread 7 state: waiting for completed aio requests (read thread) I/O thread 8 state: waiting for completed aio requests (read thread) I/O thread 9 state: waiting for completed aio requests (read thread) I/O thread 10 state: waiting for completed aio requests (write thread) I/O thread 11 state: waiting for completed aio requests (write thread) I/O thread 12 state: waiting for completed aio requests (write thread) I/O thread 13 state: waiting for completed aio requests (write thread) I/O thread 14 state: waiting for completed aio requests (write thread) I/O thread 15 state: waiting for completed aio requests (write thread) I/O thread 16 state: waiting for completed aio requests (write thread) I/O thread 17 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0, 0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 3207133 OS file reads, 4384051 OS file writes, 153435 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 18200, seg size 18202, 96998 merges merged operations: insert 661141, delete mark 1535, delete 1 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 122997258092 Log flushed up to 122997258092 Pages flushed up to 122997258092 Last checkpoint at 122997258092 Max checkpoint age 13914979615 Checkpoint age target 13480136503 Modified age 0 Checkpoint age 0 0 pending log writes, 0 pending chkp writes 26479 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 4395630592; in additional pool allocated 0 Total memory allocated by read views 408 Internal hash tables (constant factor + variable factor) Adaptive hash index 1099303648 (75692648 + 1023611000) Page hash 1107208 (buffer pool 0 only) Dictionary cache 17844813 (17702416 + 142397) File system 820296 (812272 + 8024) Lock system 37186320 (10625704 + 26560616) Recovery system 0 (0 + 0) Dictionary memory allocated 142397 Buffer pool size 262140 Buffer pool size, bytes 4294901760 Free buffers 4130 Database pages 193914 Old database pages 71507 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 2344243, not young 1062049864 0.00 youngs/s, 0.00 non-youngs/s Pages read 3206982, created 1336978, written 4249763 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 193914, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 1029 Database pages 48505 Old database pages 17891 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 613159, not young 264979076 0.00 youngs/s, 0.00 non-youngs/s Pages read 804990, created 335156, written 1090004 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 48505, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 1053 Database pages 48473 Old database pages 17873 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 591572, not young 268467481 0.00 youngs/s, 0.00 non-youngs/s Pages read 815657, created 331848, written 1054366 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 48473, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 1024 Database pages 48458 Old database pages 17868 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 596914, not young 266917822 0.00 youngs/s, 0.00 non-youngs/s Pages read 795722, created 334975, written 1042430 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 48458, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 1024 Database pages 48478 Old database pages 17875 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 542598, not young 261685485 0.00 youngs/s, 0.00 non-youngs/s Pages read 790613, created 334999, written 1062963 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 48478, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 3 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 3 out of 1000 descriptors used Main thread process no. 195615, id 139691570734848, state: sleeping Number of rows inserted 120943906, updated 1048779, deleted 3, read 516048101 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s FAIL TO OBTAIN LOCK MUTEX, SKIP LOCK INFO PRINTING ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 2014-01-20 16:30:58 7f0c7b73c700 InnoDB: Assertion failure in thread 139691587520256 in file srv0srv.cc line 2156 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 15:30:58 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster key_buffer_size=134217728 read_buffer_size=2097152 max_used_connections=16 max_threads=102 thread_count=12 connection_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759274 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8e91ae] /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x68c1e4] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f11ddc69cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f11dd0be425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f11dd0c1b8b] /usr/sbin/mysqld[0xa074cf] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f11ddc61e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f11dd17c3fd] You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. 140120 16:30:58 mysqld_safe Number of processes running now: 0 140120 16:30:58 mysqld_safe WSREP: not restarting wsrep node automatically 140120 16:30:58 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended