mysqld crash "InnoDB: Failing assertion: new_index"

[wojtek]

Hello,

Our production server crashed today without warning.
It is a shared database, so it's very difficult to debug which query or condition caused the crash.
Recovery went ok, no further crashes.
Server version: 5.1.47-rel11.2-log (Percona Server (GPL), 11.2 )

Regards,
Byte Tech Team

Attached
mysql-error.log + munin graphs

[wojtek (vojtekb)]

here is the error log:

100804 14:54:10 [Warning] Invalid (old?) table or database name '#sql2-419b-8d01c6'
100804 14:54:12 InnoDB: Assertion failure in thread 140058760623888 in file dict/dict0dict.c line 4854
InnoDB: Failing assertion: new_index
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
100804 14:54:12 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=317
max_threads=750
threads_connected=33
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1902241 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x7f61c246bb00
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f61f8acae88 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x29) [0x8ce629]
/usr/sbin/mysqld(handle_segfault+0x3db) [0x5c306b]
/lib/libpthread.so.0(+0xef60) [0x7f665417df60]
/lib/libc.so.6(gsignal+0x35) [0x7f6652423175]
/lib/libc.so.6(abort+0x180) [0x7f6652425f80]
/usr/sbin/mysqld() [0x83c1b3]
/usr/sbin/mysqld() [0x7c0d37]
/usr/sbin/mysqld() [0x777f1e]
/usr/sbin/mysqld(mysql_alter_table(THD*, char*, char*, st_ha_create_information*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool)+0x28e2) [0x6d5572]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0xcc0) [0x5d60b0]
/usr/sbin/mysqld(mysql_parse(THD*, char const*, unsigned int, char const**)+0x4af) [0x5dacbf]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd99) [0x5dba69]
/usr/sbin/mysqld(do_command(THD*)+0x12a) [0x5dc38a]
/usr/sbin/mysqld(handle_one_connection+0x34e) [0x5ce87e]
/lib/libpthread.so.0(+0x68ba) [0x7f66541758ba]
/lib/libc.so.6(clone+0x6d) [0x7f66524c001d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7f61f59a1db0 is an invalid pointer
thd->thread_id=9241040
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
100804 14:54:14 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3.4
100804 14:54:15 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan prog...

[Yasufumi Kinoshita (yasufumi-kinoshita)]

The assertion says....
The index for the foreign key cannot be found.

I suspect the table metadata inconsistency between datafiles and internal InnoDB,
and it was allowed to remove index which is needed for the foreign key kept.

Anyway, the information is not enough to estimate what is wrong...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2550