InnoDB: Failing assertion: !thr->is_active with debug build

[Teemu Ollakka]

Running standard sqlgen load against 2 nodes debug build caused the following crash:

2013-09-19 15:17:55 7fa06654f700 InnoDB: Assertion failure in thread 140326888339200 in file lock0wait.cc line 247
InnoDB: Failing assertion: !thr->is_active
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:17:55 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=12
max_threads=1024
thread_count=7
connection_count=7
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 415800 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2521100
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa06654ee40 thread_stack 0x40000
/run/shm/galera/local3/mysql/sbin/mysqld(my_print_stacktrace+0x35)[0xa9123a]
/run/shm/galera/local3/mysql/sbin/mysqld(handle_fatal_signal+0x3ae)[0x7228e6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fa095844cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fa094c9a425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7fa094c9db8b]
/run/shm/galera/local3/mysql/sbin/mysqld[0xb0c224]
/run/shm/galera/local3/mysql/sbin/mysqld[0xb98ffb]
/run/shm/galera/local3/mysql/sbin/mysqld[0xbc7b55]
/run/shm/galera/local3/mysql/sbin/mysqld[0xab173b]
/run/shm/galera/local3/mysql/sbin/mysqld[0xab1957]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN7handler13ha_index_nextEPh+0x148)[0x63ba20]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN7handler15read_range_nextEv+0x9b)[0x643611]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x7d)[0x64178f]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN10DsMrr_impl10dsmrr_nextEPPc+0x3a)[0x642614]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN11ha_innobase21multi_range_read_nextEPPc+0x2a)[0xac1bd0]
/run/shm/galera/local3/mysql/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0xa2)[0x95cf82]
/run/shm/galera/local3/mysql/sbin/mysqld[0x976190]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x16d1)[0x85f361]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2a71)[0x7ce981]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x40c)[0x7d815d]
/run/shm/galera/local3/mysql/sbin/mysqld[0x7d77b9]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1082)[0x7c9b8f]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z10do_commandP3THD+0x639)[0x7c86a4]
/run/shm/galera/local3/mysql/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1a5)[0x79215a]
/run/shm/galera/local3/mysql/sbin/mysqld(handle_one_connection+0x33)[0x791b6c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fa09583ce9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fa094d57ccd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa048146640): UPDATE comm03 SET x = (x + 1) % 65537, y = ((y*x % 65537) + (51869*y % 65537) + (y*z % 65537)) % 65537, z = ((z*x % 65537) + (z*y % 65537) + (51869*z % 65537)) % 65537, r = RAND() WHERE r > 0.0839327 AND r < 0.583933
Connection ID (thread ID): 10453
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file

Full backtrace:
#0 __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
#1 0x0000000000a912bc in my_write_core (sig=6) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/mysys/stacktrace.c:422
#2 0x0000000000722acf in handle_fatal_signal (sig=6)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/signal_handler.cc:248
#3 <signal handler called>
#4 0x00007fa094c9a425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#5 0x00007fa094c9db8b in __GI_abort () at abort.c:91
#6 0x0000000000b0c224 in lock_wait_suspend_thread (thr=0x7fa07c084a20)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/lock/lock0wait.cc:247
#7 0x0000000000b98ffb in row_mysql_handle_errors (new_err=0x7fa06654b7ec, trx=0x7fa0481751f8, thr=0x7fa07c084a20, savept=0x0)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/row/row0mysql.cc:630
#8 0x0000000000bc7b55 in row_search_for_mysql (buf=0x7fa07ca1ea80 "\341\003", mode=1, prebuilt=0x7fa07c083f78, match_mode=0,
    direction=1) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/row/row0sel.cc:5015
#9 0x0000000000ab173b in ha_innobase::general_fetch (this=0x7fa07c512780, buf=0x7fa07ca1ea80 "\341\003", direction=1, match_mode=0)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/handler/ha_innodb.cc:8280
#10 0x0000000000ab1957 in ha_innobase::index_next (this=0x7fa07c512780, buf=0x7fa07ca1ea80 "\341\003")
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/handler/ha_innodb.cc:8342
#11 0x000000000063ba20 in handler::ha_index_next (this=0x7fa07c512780, buf=0x7fa07ca1ea80 "\341\003")
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/handler.cc:2833
#12 0x0000000000643611 in handler::read_range_next (this=0x7fa07c512780)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/handler.cc:6755
#13 0x000000000064178f in handler::multi_range_read_next (this=0x7fa07c512780, range_info=0x7fa06654bf50)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/handler.cc:5858
#14 0x0000000000642614 in DsMrr_impl::dsmrr_next (this=0x7fa07c514248, range_info=0x7fa06654bf50)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/handler.cc:6238
#15 0x0000000000ac1bd0 in ha_innobase::multi_range_read_next (this=0x7fa07c512780, range_info=0x7fa06654bf50)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/storage/innobase/handler/ha_innodb.cc:17897
#16 0x000000000095cf82 in QUICK_RANGE_SELECT::get_next (this=0x7fa0480791a0)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/opt_range.cc:10585
#17 0x0000000000976190 in rr_quick (info=0x7fa06654c020) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/records.cc:367
#18 0x000000000085f361 in mysql_update (thd=0x2521100, table_list=0x7fa0481468a8, fields=..., values=..., conds=0x7fa0481043a8,
    order_num=0, order=0x0, limit=18446744073709551614, handle_duplicates=DUP_ERROR, ignore=false, found_return=0x7fa06654ce50,
    updated_return=0x7fa06654ce58) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_update.cc:633
#19 0x00000000007ce981 in mysql_execute_command (thd=0x2521100)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_parse.cc:3578
#20 0x00000000007d815d in mysql_parse (thd=0x2521100,
    rawbuf=0x7fa048146640 "UPDATE comm03 SET x = (x + 1) % 65537, y = ((y*x % 65537) + (51869*y % 65537) + (y*z % 65537)) % 65537, z = ((z*x % 65537) + (z*y % 65537) + (51869*z % 65537)) % 65537, r = RAND() WHERE r > 0.0839327 "..., length=216,
    parser_state=0x7fa06654e170) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_parse.cc:6770
#21 0x00000000007d77b9 in wsrep_mysql_parse (thd=0x2521100,
    rawbuf=0x7fa048146640 "UPDATE comm03 SET x = (x + 1) % 65537, y = ((y*x % 65537) + (51869*y % 65537) + (y*z % 65537)) % 65537, z = ((z*x % 65537) + (z*y % 65537) + (51869*z % 65537)) % 65537, r = RAND() WHERE r > 0.0839327 "..., length=216,
    parser_state=0x7fa06654e170) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_parse.cc:6551
#22 0x00000000007c9b8f in dispatch_command (command=COM_QUERY, thd=0x2521100,
    packet=0x252cc01 "UPDATE comm03 SET x = (x + 1) % 65537, y = ((y*x % 65537) + (51869*y % 65537) + (y*z % 65537)) % 65537, z = ((z*x % 65537) + (z*y % 65537) + (51869*z % 65537)) % 65537, r = RAND() WHERE r > 0.0839327 "..., packet_length=216)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_parse.cc:1532
#23 0x00000000007c86a4 in do_command (thd=0x2521100) at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_parse.cc:1126
#24 0x000000000079215a in do_handle_one_connection (thd_arg=0x2521100)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_connect.cc:1002
#25 0x0000000000791b6c in handle_one_connection (arg=0x2521100)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.6/sql/sql_connect.cc:911
#26 0x00007fa09583ce9a in start_thread (arg=0x7fa06654f700) at pthread_create.c:308
#27 0x00007fa094d57ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#28 0x0000000000000000 in ?? ()

[Seppo Jaakola (seppo-jaakola)]

Cannot reproduce this after the fix for lp:1233383

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-976