Crash in InnodbTrxTool::Generator::populate_innodb_locks (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:269

[Patrick Crews]

Crashing bug occurring during a 2 connection test load.
Both connections run via the randgen.
Connection 1: Running new data_dictionary_concurrent_drizzle.yy grammar - runs various queries against D_D tables
Connection 2: Running optimizer_subquery_drizzle.yy to generate a background load.

Crash condition:
Connection1 query: SELECT * FROM `data_dictionary` . INNODB_LOCKS
Connection2 query: SELECT table2 . `col_char_10_not_null_key` AS field1 , table1 . `col_char_10_not_null_key` AS field2 , table2 . `col_enum_not_null_key` AS field3 , table2 . `col_char_10_key` AS field4 FROM ( ( SELECT SUBQUERY1_t1 . * FROM ( AA AS SUBQUERY1_t1 STRAIGHT_JOIN ( BB AS SUBQUERY1_t2 RIGHT JOIN CC AS SUBQUERY1_t3 ON (SUBQUERY1_t3 . `col_char_1024_key` = SUBQUERY1_t2 . `col_text_not_null` ) ) ON (SUBQUERY1_t3 . `col_bigint_key` = SUBQUERY1_t2 . `col_bigint_key` OR SUBQUERY1_t1 . `col_char_10_not_null` < ANY ( SELECT 3 UNION SELECT 2 ) ) ) ) AS table1 STRAIGHT_JOIN ( ( ( SELECT SQL_SMALL_RESULT SUBQUERY2_t1 . * FROM ( AA AS SUBQUERY2_t1 LEFT JOIN CC AS SUBQUERY2_t2 ON (SUBQUERY2_t2 . `col_bigint` = SUBQUERY2_t1 . `pk` ) ) ) AS table2 INNER JOIN ( SELECT SUBQUERY3_t1 . * FROM ( C AS SUBQUERY3_t1 LEFT JOIN CC AS SUBQUERY3_t2 ON (SUBQUERY3_t2 . `col_int_not_null` = SUBQUERY3_t1 . `col_int_not_null_key` ) ) ) AS table3 ON (table3 . `col_char_10_not_null_key` = table2 . `col_char_10_key` ) ) ) ON (table3 . `col_char_1024_not_null_key` = table2 . `col_char_1024_key` ) ) WHERE ( NOT EXISTS ( SELECT SUBQUERY4_t1 . `col_char_1024_not_null` AS SUBQUERY4_field1 FROM ( BB AS SUBQUERY4_t1 STRAIGHT_JOIN BB AS SUBQUERY4_t2 ON (SUBQUERY4_t2 . `col_char_1024_not_null_key` = SUBQUERY4_t1 . `col_text_key` ) ) WHERE SUBQUERY4_t1 . `col_char_1024` > table2 . `col_text_key` AND SUBQUERY4_t2 . `col_char_1024_not_null_key` < 'o' ) ) AND ( table1 . `col_int_not_null_key` = 99 OR table1 . `col_bigint_key` = table2 . `col_bigint_key` ) ORDER BY table1 . `col_enum_not_null_key` ASC

[Patrick Crews (patrick-crews)]

Backtrace:

#0 0x00007f26345f7a75 in *__GI_raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f26345fb5c0 in *__GI_abort () at abort.c:92
#2 0x00000000007bd29f in trx_i_s_cache_get_nth_row (cache=<value optimized out>, table=<value optimized out>, n=<value optimized out>)
    at plugin/innobase/trx/trx0i_s.c:1392
#3 0x000000000070b7e4 in InnodbTrxTool::Generator::populate_innodb_locks (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:269
#4 0x000000000070b9d5 in InnodbTrxTool::Generator::populate (this=0x7f26140046f0) at plugin/innobase/handler/data_dictionary.cc:231
#5 0x00000000005ed7d1 in drizzled::plugin::TableFunction::Generator::sub_populate (this=0x255, field_size=23321) at drizzled/plugin/table_function.cc:156
#6 0x00000000006d6b4e in FunctionCursor::rnd_next (this=0x7f2614105cf0) at plugin/function_engine/cursor.cc:87
#7 0x00000000005f1bd1 in rr_sequential (info=0x7f2614022948) at drizzled/records.cc:271
#8 0x0000000000629331 in drizzled::sub_select (join=0x34bbf98, join_tab=0x5b19, end_of_records=6) at drizzled/sql_select.cc:3563
#9 0x000000000062e330 in drizzled::do_select (join=0x34bbf98, fields=0x7f25dc086bb0, table=0x0) at drizzled/sql_select.cc:3333
#10 0x00000000005a406c in drizzled::Join::exec (this=0x34bbf98) at drizzled/join.cc:1695
#11 0x000000000062a6a2 in drizzled::mysql_select (session=0x7f25dc085e40, rref_pointer_array=<value optimized out>, tables=0x34bbdd0, wild_num=1,
    fields=<value optimized out>, conds=<value optimized out>, og_num=0, order=0x0, group=0x0, having=0x0, select_options=2147500032, result=0x34bbf78,
    unit=0x7f25dc0868b8, select_lex=0x7f25dc086ab8) at drizzled/sql_select.cc:427
#12 0x000000000062afa0 in drizzled::handle_select (session=0x7f25dc085e40, lex=0x7f25dc086898, result=0x34bbf78, setup_tables_done_option=0)
    at drizzled/sql_select.cc:146
#13 0x0000000000624e23 in drizzled::execute_sqlcom_select (session=0x7f25dc085e40, all_tables=0x34bbdd0) at drizzled/sql_parse.cc:544
#14 0x0000000000624a3e in mysql_execute_command (session=0x7f25dc085e40, inBuf=<value optimized out>, length=<value optimized out>)
    at drizzled/sql_parse.cc:479
#15 drizzled::mysql_parse (session=0x7f25dc085e40, inBuf=<value optimized out>, length=<value optimized out>) at drizzled/sql_parse.cc:750
#16 0x0000000000626e12 in drizzled::dispatch_command (command=drizzled::COM_QUERY, session=0x7f25dc085e40, packet=0x7f25dc041aa1 "",
    packet_length=<value optimized out>) at drizzled/sql_parse.cc:224
#17 0x00000000005f508a in drizzled::Session::executeStatement (this=0x7f25dc085e40) at drizzled/session.cc:666
#18 0x00000000005f8132 in drizzled::Session::run (this=0x7f25dc085e40) at drizzled/session.cc:522
#19 0x00007f2630d65344 in MultiThreadScheduler::runSession (arg=<value optimized out>) at ./plugin/multi_thread/multi_thread.h:67
#20 session_thread (arg=<value optimized out>) at plugin/multi_thread/multi_thread.cc:52
#21 0x00007f263494d9ca in start_thread (arg=<value optimized out>) at pthread_create.c:300
#22 0x00007f26346aa6fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#23 0x0000000000000000 in ?? ()

[Patrick Crews (patrick-crews)]

To repeat:

Start a drizzle server: ./test-run --start-and-exit
have two randgen command lines.
command line 1: ./gentest.pl --dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test --threads=1 --engine=Innodb --gendata=conf/drizzle/drizzle.zz --grammar=conf/drizzle/data_dict_concurrent_drizzle.yy --debug --queries=100000 --threads=1

command line 2: ./gentest.pl --dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test --threads=1 --engine=Innodb --grammar=conf/drizzle/optimizer_subquery_drizzle.yy --debug --queries=1000

Start command line 1 and let it run for a bit, before initiating command line 2 - you should see this crash before long

[Patrick Crews (patrick-crews)]

It appears to be crashing when someone tries to examine the innodb_locks table at the same time it is being updated.

[Andrew Hutchings (linuxjedi)]

this is one I'm pretty sure I fixed a while ago, and I can no longer reproduce it.