MySQL server got assertion failure

[Andrey]

MySQL server got assertion failure

121224 2:45:49 InnoDB: Assertion failure in thread 47020297689408 in file fsp0fsp.c line 3343
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:45:49 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=4294967296
read_buffer_size=2097152
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13416432 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a8185]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x683e84]
/lib64/libpthread.so.0[0x336fe0ebe0]
/lib64/libc.so.6(gsignal+0x35)[0x336ea30285]
/lib64/libc.so.6(abort+0x110)[0x336ea31d30]
/usr/sbin/mysqld[0x888d61]
/usr/sbin/mysqld[0x889128]
/usr/sbin/mysqld[0x82a898]
/usr/sbin/mysqld[0x8e5e63]
/usr/sbin/mysqld[0x8e6838]
/usr/sbin/mysqld[0x8dad9f]
/usr/sbin/mysqld[0x7fed07]
/usr/sbin/mysqld[0x7f541c]
/lib64/libpthread.so.0[0x336fe0677d]
/lib64/libc.so.6(clone+0x6d)[0x336ead3c1d]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
121224 02:45:49 mysqld_safe mysqld from pid file /data/mysql/457891-db1.*****.com.pid ended
121224 02:46:05 mysqld_safe Starting mysqld daemon with databases from /data/mysql
InnoDB: Warning: innodb_log_block_size has been changed from default value 512. (###EXPERIMENTAL### operation)
InnoDB: The log block size is set to 4096.
121224 2:46:05 InnoDB: The InnoDB memory heap is disabled
121224 2:46:05 InnoDB: Mutexes and rw_locks use GCC atomic builtins
121224 2:46:05 InnoDB: Compressed tables use zlib 1.2.3
121224 2:46:05 InnoDB: Using Linux native AIO
121224 2:46:05 InnoDB: Error: Linux Native AIO is not supported on tmpdir.
InnoDB: You can either move tmpdir to a file system that supports native AIO
InnoDB: or you can set innodb_use_native_aio to FALSE to avoid this message.
121224 2:46:05 InnoDB: Error: Linux Native AIO check on tmpdir returned error[22]
121224 2:46:05 InnoDB: Warning: Linux Native AIO disabled.
121224 2:46:05 InnoDB: Initializing buffer pool, size = 90.0G
121224 2:46:11 InnoDB: Completed initialization of buffer pool
121224 2:46:11 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 1456676929187
121224 2:46:12 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 1456682168320
InnoDB: Doing recovery: scanned up to log sequence number 1456687411200
InnoDB: Doing recovery: scanned up to log sequence number 1456692654080
InnoDB: Doing recovery: scanned up to log sequence number 1456697896960
InnoDB: Doing recovery: scanned up to log sequence number 1456703139840
InnoDB: Doing recovery: scanned up to log sequence number 1456708382720
InnoDB: Doing recovery: scanned up to log sequence number 1456713625600
InnoDB: Doing recovery: scanned up to log sequence number 1456718868480
InnoDB: Doing recovery: scanned up to log sequence number 1456724111360
InnoDB: Doing recovery: scanned up to log sequence number 1456729354240
InnoDB: Doing recovery: scanned up to log sequence number 1456734597120
InnoDB: Doing recovery: scanned up to log sequence number 1456739840000
InnoDB: Doing recovery: scanned up to log sequence number 1456745082880
InnoDB: Doing recovery: scanned up to log sequence number 1456750325760
InnoDB: Doing recovery: scanned up to log sequence number 1456755568640
InnoDB: Doing recovery: scanned up to log sequence number 1456760811520
InnoDB: Doing recovery: scanned up to log sequence number 1456766054400
InnoDB: Doing recovery: scanned up to log sequence number 1456771297280
InnoDB: Doing recovery: scanned up to log sequence number 1456776540160
InnoDB: Doing recovery: scanned up to log sequence number 1456781783040
InnoDB: Doing recovery: scanned up to log sequence number 1456787025920
InnoDB: Doing recovery: scanned up to log sequence number 1456792268800
InnoDB: Doing recovery: scanned up to log sequence number 1456797486104
121224 2:46:15 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 834732791, file name db9-457892-bin-log.002524
InnoDB: and relay log file
InnoDB: position 358977, file name /var/lib/mysqllogs/db8-457891-relay-log.000434
InnoDB: Last MySQL binlog file position 0 316448305, file name /var/lib/mysqllogs/db8-457891-bin-log.002619
121224 2:46:23 InnoDB: Waiting for the background threads to start
InnoDB: Dump of the tablespace extent descriptor: len 40; hex 0000000000000002000bc00017be000bc00017e600000004aaaaaaaaaaaaaafaffaaaaaaaaaaaaaa; asc ;
InnoDB: Serious error! InnoDB is trying to free page 502814
InnoDB: though it is already marked as free in the tablespace!
InnoDB: The tablespace free space info is corrupt.
InnoDB: You may need to dump your InnoDB tables and recreate the whole
InnoDB: database!
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
121224 2:46:23 InnoDB: Assertion failure in thread 47020205410624 in file fsp0fsp.c line 3343
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
08:46:23 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=4294967296
read_buffer_size=2097152
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13416432 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7a8185]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x683e84]
/lib64/libpthread.so.0[0x336fe0ebe0]
/lib64/libc.so.6(gsignal+0x35)[0x336ea30285]
/lib64/libc.so.6(abort+0x110)[0x336ea31d30]
/usr/sbin/mysqld[0x888d61]
/usr/sbin/mysqld[0x889128]
/usr/sbin/mysqld[0x82a898]
/usr/sbin/mysqld[0x8e5e63]
/usr/sbin/mysqld[0x8e6838]
/usr/sbin/mysqld[0x8dad9f]
/usr/sbin/mysqld[0x7fed07]
/usr/sbin/mysqld[0x7f541c]
/lib64/libpthread.so.0[0x336fe0677d]
/lib64/libc.so.6(clone+0x6d)[0x336ead3c1d]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
121224 02:46:24 mysqld_safe mysqld from pid file /data/mysql/457891-db1.****.com.pid ended

[Raghavendra D Prabhu (raghavendra-prabhu)]

The crash seems to be in fseg_free_page_low and is happening (as
the error suggests) because it is trying to free a page which is
already marked free. This is possibly due to corruption in your
tablespace.

Can you try to dump the table and reload the tables with
innodb_force_recovery?

[Valerii Kravchuk (valerii-kravchuk)]

What exact version of Percona Server, 5.5.x, it was? Had you tried to dump and reload the table as suggested by Raghavendra?

See also upstream http://bugs.mysql.com/bug.php?id=68302.