Complete error log from the instance, based on all files I could find still on system. Note I deleted details from the actual page dump as that leaks information about contents of the page. $ cat error.log.3 error.log.2 error.log.1 error.log 150302 20:06:14 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150302 20:06:14 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. 150302 20:06:14 InnoDB: The InnoDB memory heap is disabled 150302 20:06:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150302 20:06:14 InnoDB: Compressed tables use zlib 1.2.3.4 150302 20:06:14 InnoDB: Using Linux native AIO 150302 20:06:15 InnoDB: Initializing buffer pool, size = 196.5G 150302 20:06:27 InnoDB: Completed initialization of buffer pool 150302 20:06:27 InnoDB: highest supported file format is Barracuda. 150302 20:06:36 InnoDB: Waiting for the background threads to start 150302 20:06:37 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 1597945 InnoDB: something wrong with relay-log.info. InnoDB will not overwrite it. 150302 20:06:38 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode. 150302 20:06:38 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150302 20:06:38 [Note] Event Scheduler: Loaded 0 events 150302 20:06:38 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 150302 20:07:06 [Note] /usr/sbin/mysqld: Normal shutdown 150302 20:07:06 [Note] Event Scheduler: Purging the queue. 0 events 150302 20:07:06 InnoDB: Starting shutdown... 150302 20:07:14 InnoDB: Shutdown completed; log sequence number 1597945 150302 20:07:14 [Note] /usr/sbin/mysqld: Shutdown complete 150302 20:27:55 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150302 20:27:55 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. 150302 20:27:55 InnoDB: The InnoDB memory heap is disabled 150302 20:27:55 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150302 20:27:55 InnoDB: Compressed tables use zlib 1.2.3.4 150302 20:27:55 InnoDB: Using Linux native AIO 150302 20:27:55 InnoDB: Initializing buffer pool, size = 196.5G 150302 20:28:07 InnoDB: Completed initialization of buffer pool 150302 20:28:07 InnoDB: highest supported file format is Barracuda. 150302 20:28:17 InnoDB: Waiting for the background threads to start 150302 20:28:18 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10279806215055 InnoDB: something wrong with relay-log.info. InnoDB will not overwrite it. 150302 20:28:18 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode. 150302 20:28:18 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150302 20:28:18 [Note] Event Scheduler: Loaded 0 events 150302 20:28:18 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 150302 20:39:35 [Note] 'CHANGE MASTER TO executed'. Previous state master_host='', master_port='3306', master_log_file='', master_log_pos='4'. New state master_host='172.19.51.61', master_port='3306', master_log_file='bin.000131', master_log_pos='934527426'. 150302 20:39:37 [Note] Slave SQL thread initialized, starting replication in log 'bin.000131' at position 934527426, relay log './local-relay-bin.000001' position: 4 150302 20:39:37 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000131' at position 934527426 150304 18:36:27 InnoDB: Assertion failure in thread 140331370538752 in file btr0cur.c line 329 InnoDB: Failing assertion: btr_page_get_prev(get_block->frame, mtr) == page_get_page_no(page) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 20:35:48 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=4 max_threads=16386 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fd4f22e2fa9] /usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7fd4f21c8950] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fd4f1373cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fd4efd5f425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fd4efd62b8b] /usr/sbin/mysqld(+0x5a0d85)[0x7fd4f2367d85] /usr/sbin/mysqld(+0x5a1e2b)[0x7fd4f2368e2b] /usr/sbin/mysqld(+0x5ac12c)[0x7fd4f237312c] /usr/sbin/mysqld(+0x659472)[0x7fd4f2420472] /usr/sbin/mysqld(+0x65aa90)[0x7fd4f2421a90] /usr/sbin/mysqld(+0x650ae5)[0x7fd4f2417ae5] /usr/sbin/mysqld(+0x57c52b)[0x7fd4f234352b] /usr/sbin/mysqld(+0x571b0c)[0x7fd4f2338b0c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd4f136be9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd4efe1d3fd] You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. Writing a core file InnoDB: Warning: a long semaphore wait: 150304 21:05:53 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150304 21:05:53 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. InnoDB: relay-log.info is detected. InnoDB: relay log: position 585677554, file name ./local-relay-bin.000024 InnoDB: master log: position 585677414, file name bin.000139 150304 21:05:53 InnoDB: The InnoDB memory heap is disabled 150304 21:05:53 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150304 21:05:53 InnoDB: Compressed tables use zlib 1.2.3.4 150304 21:05:53 InnoDB: Using Linux native AIO 150304 21:05:54 InnoDB: Initializing buffer pool, size = 196.5G 150304 21:06:06 InnoDB: Completed initialization of buffer pool 150304 21:06:06 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 10295229374596 150304 21:06:08 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 10295229695015 150304 21:06:10 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 585677414, file name bin.000139 InnoDB: and relay log file InnoDB: position 585677554, file name ./local-relay-bin.000024 InnoDB: Last MySQL binlog file position 0 494719823, file name /data/mysql_binlog/bin.000009 150304 21:06:20 InnoDB: Waiting for the background threads to start 150304 21:06:21 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10295229695015 InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it. 150304 21:06:21 [Note] Recovering after a crash using /data/mysql_binlog/bin 150304 21:06:24 [Note] Starting crash recovery... 150304 21:06:24 [Note] Crash recovery finished. 150304 21:06:24 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode. 150304 21:06:24 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150304 21:06:24 [Note] Slave SQL thread initialized, starting replication in log 'bin.000139' at position 585677414, relay log './local-relay-bin.000024' position: 585677554 150304 21:06:24 [Note] Event Scheduler: Loaded 0 events 150304 21:06:24 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000139' at position 585700509 150304 21:06:24 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 150304 21:13:19 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 838:0, should be 838:111751! InnoDB: N heap is wrong 109, 96 150305 4:16:41 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex ... data deleted here ... InnoDB: End of page dump 150305 4:16:41 InnoDB: Page checksum 2449463521 (32bit_calc: 3477942518), prior-to-4.0.14-form checksum 3461508486 InnoDB: stored checksum 2449463521, prior-to-4.0.14-form stored checksum 3461508486 InnoDB: Page lsn 2397 3992988175, low 4 bytes of lsn at page end 3992988175 InnoDB: Page number (if stored to page already) 347275, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 838 InnoDB: Page may be an index page where index id is 2016 InnoDB: (index "GEN_CLUST_INDEX" of table "ejabberd_ody_mobboss_online"."rosterusers") 150305 4:16:41 InnoDB: Apparent corruption of an index page n:o 347275 in space 838 InnoDB: to be written to data file. We intentionally crash server InnoDB: to prevent corrupt data from ending up in data InnoDB: files. 150305 4:16:41 InnoDB: Assertion failure in thread 139545601242880 in file buf0flu.c line 806 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 04:16:41 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=3 max_threads=16386 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f1dfce06fa9] /usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7f1dfccec950] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f1dfbe97cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f1dfa883425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f1dfa886b8b] /usr/sbin/mysqld(+0x5bfe48)[0x7f1dfceaae48] /usr/sbin/mysqld(+0x5c2c15)[0x7f1dfceadc15] /usr/sbin/mysqld(+0x5c311c)[0x7f1dfceae11c] /usr/sbin/mysqld(+0x5c35c2)[0x7f1dfceae5c2] /usr/sbin/mysqld(+0x571368)[0x7f1dfce5c368] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f1dfbe8fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1dfa9413fd] You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. Writing a core file 150305 4:17:35 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150305 4:17:35 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. InnoDB: relay-log.info is detected. InnoDB: relay log: position 370401210, file name ./local-relay-bin.000031 InnoDB: master log: position 370401070, file name bin.000141 150305 4:17:35 InnoDB: The InnoDB memory heap is disabled 150305 4:17:35 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150305 4:17:35 InnoDB: Compressed tables use zlib 1.2.3.4 150305 4:17:35 InnoDB: Using Linux native AIO 150305 4:17:35 InnoDB: Initializing buffer pool, size = 196.5G 150305 4:17:47 InnoDB: Completed initialization of buffer pool 150305 4:17:47 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 10299028323666 150305 4:17:49 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 10299029807255 150305 4:17:53 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 370401070, file name bin.000141 InnoDB: and relay log file InnoDB: position 370401210, file name ./local-relay-bin.000031 InnoDB: Last MySQL binlog file position 0 754983375, file name /data/mysql_binlog/bin.000011 150305 4:18:18 InnoDB: Waiting for the background threads to start 150305 4:18:19 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10299029807255 InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it. 150305 4:18:19 [Note] Recovering after a crash using /data/mysql_binlog/bin 150305 4:18:22 [Note] Starting crash recovery... 150305 4:18:22 [Note] Crash recovery finished. 150305 4:19:02 [Warning] 'user' entry 'root@XXX ignored in --skip-name-resolve mode. 150305 4:19:02 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150305 4:19:12 [Note] Slave SQL thread initialized, starting replication in log 'bin.000141' at position 370401070, relay log './local-relay-bin.000031' position: 370401210 150305 4:19:12 [Note] Event Scheduler: Loaded 0 events 150305 4:19:12 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 150305 4:19:12 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000141' at position 370408948 150305 4:24:51 InnoDB: Error: space id and page n:o stored in the page InnoDB: read in are 838:0, should be 838:111751! InnoDB: Next record offset nonsensical 0 for rec 4108 150305 7:32:52 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex ... data deleted here ... InnoDB: End of page dump 150305 7:32:52 InnoDB: Page checksum 578674375 (32bit_calc: 1594542261), prior-to-4.0.14-form checksum 1637013161 InnoDB: stored checksum 578674375, prior-to-4.0.14-form stored checksum 1637013161 InnoDB: Page lsn 2398 715216317, low 4 bytes of lsn at page end 715216317 InnoDB: Page number (if stored to page already) 166019, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 837 InnoDB: Page may be an index page where index id is 2015 InnoDB: (index "pk_rosterg_user_jid" of table "ejabberd_ody_mobboss_online"."rostergroups") 150305 7:32:52 InnoDB: Apparent corruption of an index page n:o 166019 in space 837 InnoDB: to be written to data file. We intentionally crash server InnoDB: to prevent corrupt data from ending up in data InnoDB: files. 150305 7:32:52 InnoDB: Assertion failure in thread 139654153164544 in file buf0flu.c line 806 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 07:32:52 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=4 max_threads=16386 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7f3742cc8fa9] /usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7f3742bae950] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f3741d59cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f3740745425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f3740748b8b] /usr/sbin/mysqld(+0x5bfe48)[0x7f3742d6ce48] /usr/sbin/mysqld(+0x5c2c15)[0x7f3742d6fc15] /usr/sbin/mysqld(+0x5c311c)[0x7f3742d7011c] /usr/sbin/mysqld(+0x5c35c2)[0x7f3742d705c2] /usr/sbin/mysqld(+0x571368)[0x7f3742d1e368] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f3741d51e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f37408033fd] You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. Writing a core file 150305 7:33:33 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150305 7:33:33 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. InnoDB: relay-log.info is detected. InnoDB: relay log: position 147640616, file name ./local-relay-bin.000036 InnoDB: master log: position 147640513, file name bin.000142 150305 7:33:33 InnoDB: The InnoDB memory heap is disabled 150305 7:33:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150305 7:33:33 InnoDB: Compressed tables use zlib 1.2.3.4 150305 7:33:33 InnoDB: Using Linux native AIO 150305 7:33:33 InnoDB: Initializing buffer pool, size = 196.5G 150305 7:33:45 InnoDB: Completed initialization of buffer pool 150305 7:33:46 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 10300046723171 150305 7:33:48 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 10300047265987 150305 7:33:59 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 147640513, file name bin.000142 InnoDB: and relay log file InnoDB: position 147640616, file name ./local-relay-bin.000036 InnoDB: Last MySQL binlog file position 0 139653441, file name /data/mysql_binlog/bin.000013 150305 7:34:24 InnoDB: Waiting for the background threads to start 150305 7:34:25 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10300047265987 InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it. 150305 7:34:25 [Note] Recovering after a crash using /data/mysql_binlog/bin 150305 7:34:26 [Note] Starting crash recovery... 150305 7:34:26 [Note] Crash recovery finished. 150305 7:34:40 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode. 150305 7:34:40 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150305 7:34:44 [Note] Slave SQL thread initialized, starting replication in log 'bin.000142' at position 147640513, relay log './local-relay-bin.000036' position: 147640616 150305 7:34:44 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000142' at position 147640513 150305 7:34:44 [Note] Event Scheduler: Loaded 0 events 150305 7:34:44 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 InnoDB: Record 13110 is above rec heap top 12973 150305 10:36:48 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex ... data truncated here ... InnoDB: End of page dump 150305 10:36:48 InnoDB: Page checksum 2692330666 (32bit_calc: 2825342776), prior-to-4.0.14-form checksum 1830467664 InnoDB: stored checksum 2692330666, prior-to-4.0.14-form stored checksum 1830467664 InnoDB: Page lsn 2398 1288429380, low 4 bytes of lsn at page end 1288429380 InnoDB: Page number (if stored to page already) 12766, InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 821 InnoDB: Page may be an index page where index id is 1983 InnoDB: (index "uniq" of table "ejabberd_ody_mobboss_online"."muc_affiliations") 150305 10:36:48 InnoDB: Apparent corruption of an index page n:o 12766 in space 821 InnoDB: to be written to data file. We intentionally crash server InnoDB: to prevent corrupt data from ending up in data InnoDB: files. 150305 10:36:48 InnoDB: Assertion failure in thread 140258901358336 in file buf0flu.c line 806 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 10:36:48 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=67108864 read_buffer_size=131072 max_used_connections=4 max_threads=16386 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 35924364 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x29)[0x7fc410bd9fa9] /usr/sbin/mysqld(handle_fatal_signal+0x3f0)[0x7fc410abf950] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fc40fc6acb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fc40e656425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fc40e659b8b] /usr/sbin/mysqld(+0x5bfe48)[0x7fc410c7de48] /usr/sbin/mysqld(+0x5c2c15)[0x7fc410c80c15] /usr/sbin/mysqld(+0x5c311c)[0x7fc410c8111c] /usr/sbin/mysqld(+0x5c35c2)[0x7fc410c815c2] /usr/sbin/mysqld(+0x571368)[0x7fc410c2f368] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc40fc62e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc40e7143fd] You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash. Writing a core file 150305 10:37:23 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150305 10:37:23 [Note] Plugin 'FEDERATED' is disabled. InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized. InnoDB: relay-log.info is detected. InnoDB: relay log: position 285907453, file name ./local-relay-bin.000038 InnoDB: master log: position 433547719, file name bin.000142 150305 10:37:23 InnoDB: The InnoDB memory heap is disabled 150305 10:37:23 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150305 10:37:23 InnoDB: Compressed tables use zlib 1.2.3.4 150305 10:37:23 InnoDB: Using Linux native AIO 150305 10:37:23 InnoDB: Initializing buffer pool, size = 196.5G 150305 10:37:35 InnoDB: Completed initialization of buffer pool 150305 10:37:36 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 10300619805596 150305 10:37:38 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 10300620307881 150305 10:37:48 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 433547719, file name bin.000142 InnoDB: and relay log file InnoDB: position 285907453, file name ./local-relay-bin.000038 InnoDB: Last MySQL binlog file position 0 270154769, file name /data/mysql_binlog/bin.000014 150305 10:38:08 InnoDB: Waiting for the background threads to start 150305 10:38:09 Percona XtraDB (http://www.percona.com) 5.5.41-37.0 started; log sequence number 10300620307881 InnoDB: InnoDB and relay-log.info are synchronized. InnoDB will not overwrite it. 150305 10:38:09 [Note] Recovering after a crash using /data/mysql_binlog/bin 150305 10:38:10 [Note] Starting crash recovery... 150305 10:38:10 [Note] Crash recovery finished. 150305 10:38:19 [Warning] 'user' entry 'root@XXX' ignored in --skip-name-resolve mode. 150305 10:38:19 [Warning] 'proxies_priv' entry '@ root@XXX' ignored in --skip-name-resolve mode. 150305 10:38:19 [Note] Slave SQL thread initialized, starting replication in log 'bin.000142' at position 433547719, relay log './local-relay-bin.000038' position: 285907453 150305 10:38:19 [Note] Event Scheduler: Loaded 0 events 150305 10:38:19 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.41-37.0-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 37.0, Revision 727 150305 10:38:19 [Note] Slave I/O thread: connected to master 'repl@172.19.51.61:3306',replication started in log 'bin.000142' at position 433547719