random failure on startup due to innodb assertion

[Patrick Crews]

The server appears to hit an innodb assertion during startup at random times.

This appears to be tied to the percona patches introduced in revision 2274. I ran tests against a branch at revision=2273 and have yet to hit this failure.

A good way to repeat this is:
./dbqp --suite=catalog --repeat=20

This will repeat each test in the catalog suite 20x and is a good way to hit the random failure.

Error log output:
cat workdir/bot0/s0/var/log/s0.err
InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
110426 16:51:39 InnoDB: Assertion failure in thread 140448072947456 in file plugin/innobase/srv/srv0srv.cc line 899
InnoDB: Failing assertion: slot->in_use == FALSE
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110426 16:51:39 - drizzled got signal 6;
This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

read_buffer_size=131072
max_used_connections=0
connection_count=0
It is possible that drizzled could use up to
(read_buffer_size + sort_buffer_size)*thread_count
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of stack frames obtained: 9
drizzle/drizzled/.libs/lt-drizzled() [0x6b44bd]
()
gsignal()
abort()
drizzle/drizzled/.libs/lt-drizzled() [0x850252]
drizzle/drizzled/.libs/lt-drizzled() [0x852f7a]
()
clone()

[David Shrewsbury (dshrews)]

This looks just like that weird crash that I identified at the UC. That one was:

InnoDB: Doublewrite buffer not found: creating new
InnoDB: Doublewrite buffer created
InnoDB: 127 rollback segment(s) active.
InnoDB: Creating foreign key constraint system tables
InnoDB: Foreign key constraint system tables created
110412 7:57:53 InnoDB: Assertion failure in thread 4527833088 in file plugin/innobase/srv/srv0srv.cc line 899
InnoDB: Failing assertion: slot->in_use == FALSE
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110412 7:57:53 - drizzled got signal 6;
This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

read_buffer_size=131072
max_used_connections=0
connection_count=0
It is possible that drizzled could use up to
(read_buffer_size + sort_buffer_size)*thread_count
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of stack frames obtained: 15
1 drizzled 0x00000001001c3ede drizzled_handle_segfault + 455
2 libSystem.B.dylib 0x00007fff81b1166a _sigtramp + 26
3 libinnobase_plugin.dylib 0x0000000101ae18a7 _ZL15mutex_exit_funcP12mutex_struct + 30
4 libinnobase_plugin.dylib 0x0000000101b41b17 _Z15mutex_spin_waitP12mutex_structPKcm + 90
5 libinnobase_plugin.dylib 0x0000000101b54475 _ZL16mutex_enter_funcP12mutex_structPKcm + 55
6 libinnobase_plugin.dylib 0x0000000101b57ce6 _Z22trx_allocate_for_mysqlv + 35
7 libinnobase_plugin.dylib 0x0000000101a9815e _Z31dict_create_sys_replication_logv + 120
8 libinnobase_plugin.dylib 0x0000000101a123fd _ZL13innobase_initRN8drizzled6module7ContextE + 6655
9 drizzled 0x000000010012df42 _ZN8drizzledL17plugin_initializeERNS_6module8RegistryEPNS0_6ModuleE + 123
10 drizzled 0x000000010012e585 _ZN8drizzled15plugin_finalizeERNS_6module8RegistryE + 348
11 drizzled 0x0000000100018ab1 _ZN8drizzled22init_server_componentsERNS_6module8RegistryE + 166
12 drizzled 0x000000010012482f main + 1046
13 drizzled 0x00000001000015b8 start + 52
14 ??? 0x000000000000000f 0x0 + 15