InnoDB crashes with buffer pool in shared memory

[Vadim Tkachenko]

Sometimes I got crash when InnoDB starts with existing buffer pool

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x59bee940 (LWP 18353)]
ha_insert_for_fold_func (table=0x135a97a8, fold=944029509632058041, data=0x2abc0a4e9b8e) at ha/ha0ha.c:166
166 ha/ha0ha.c: No such file or directory.
        in ha/ha0ha.c

(gdb) bt
#0 ha_insert_for_fold_func (table=0x135a97a8, fold=944029509632058041, data=0x2abc0a4e9b8e) at ha/ha0ha.c:166
#1 0x00000000007d291e in btr_search_build_page_hash_index (index=0x1395b8b8, block=0x2aab17d14790,
    n_fields=<value optimized out>, n_bytes=<value optimized out>, left_side=<value optimized out>) at btr/btr0sea.c:1520
#2 0x00000000007d3419 in btr_search_info_update_slow (info=<value optimized out>, cursor=0x2aca20170ff8) at btr/btr0sea.c:641
#3 0x00000000007c7362 in btr_search_info_update (index=0x1395b8b8, level=0, tuple=0x2aca201712b8, mode=2,
    latch_mode=<value optimized out>, cursor=0x2aca20170ff8, has_search_latch=0, file=0x9b83d7 "row/row0sel.c", line=3707,
    mtr=0x59beaa40) at ./include/btr0sea.ic:83
#4 btr_cur_search_to_nth_level (index=0x1395b8b8, level=0, tuple=0x2aca201712b8, mode=2, latch_mode=<value optimized out>,
    cursor=0x2aca20170ff8, has_search_latch=0, file=0x9b83d7 "row/row0sel.c", line=3707, mtr=0x59beaa40) at btr/btr0cur.c:710
#5 0x0000000000789d68 in btr_pcur_open_with_no_init_func (buf=0x2aca20168630 "\377\377\363", <incomplete sequence \335>,
    mode=2, prebuilt=0x2aca20170d88, match_mode=1, direction=0) at ./include/btr0pcur.ic:542
#6 row_search_for_mysql (buf=0x2aca20168630 "\377\377\363", <incomplete sequence \335>, mode=2, prebuilt=0x2aca20170d88,
    match_mode=1, direction=0) at row/row0sel.c:3705
#7 0x0000000000730675 in ha_innobase::index_read (this=0x2aca2016ea50,
    buf=0x2aca20168630 "\377\377\363", <incomplete sequence \335>, key_ptr=<value optimized out>, key_len=<value optimized out>,
    find_flag=HA_READ_KEY_EXACT) at handler/ha_innodb.cc:5800
#8 0x000000000068efd9 in handler::index_read_idx_map (this=0x2aca2016ea50,
    buf=0x2aca20168630 "\377\377\363", <incomplete sequence \335>, index=<value optimized out>, key=0x13959f28 "\253\003?\r",
    keypart_map=3, find_flag=HA_READ_KEY_EXACT) at handler.cc:4428
#9 0x0000000000618f77 in join_read_const (tab=0x13959cc0) at sql_select.cc:11923
#10 0x0000000000623afd in join_read_const_table (tab=0x13959cc0, pos=0x13958230) at sql_select.cc:11826
#11 0x000000000062453c in make_join_statistics (join=0x13958198, tables_arg=0x2aca240f7050, conds=0x13957ff0,
    keyuse_array=0x13959758) at sql_select.cc:2939
#12 0x0000000000625f6f in JOIN::optimize (this=0x13958198) at sql_select.cc:1041
#13 0x000000000062e991 in mysql_select (thd=0x2aca1c0ef030, rref_pointer_array=0x2aca240f56c8, tables=0x2aca240f7050,
    wild_num=0, fields=..., conds=0x13957ff0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0,
    select_options=2416724480, result=0x2aca240f80f8, unit=0x2aca240f50d0, select_lex=0x2aca240f54f8) at sql_select.cc:2524
#14 0x000000000062f3e5 in handle_select (thd=0x2aca1c0ef030, lex=0x2aca240f5030, result=0x2aca240f80f8,
    setup_tables_done_option=0) at sql_select.cc:284
#15 0x00000000005bd141 in execute_sqlcom_select (thd=0x2aca1c0ef030, all_tables=0x2aca240f7050) at sql_parse.cc:5166
#16 0x00000000005c03c1 in mysql_execute_command (thd=0x2aca1c0ef030) at sql_parse.cc:2357
#17 0x000000000063aab0 in Prepared_statement::execute (this=0x2aca24081f00, expanded_query=<value optimized out>,
    open_cursor=false) at sql_prepare.cc:3876
#18 0x000000000063cb64 in Prepared_statement::execute_loop (this=0x2aca24081f00, expanded_query=0x59bed1a0, open_cursor=false,
    packet=<value optimized out>, packet_end=<value optimized out>) at sql_prepare.cc:3551
#19 0x000000000063ce64 in mysqld_stmt_execute (thd=0x2aca1c0ef030, packet_arg=0x2aca1c0f21b1 "\a", packet_length=23)
    at sql_prepare.cc:2598
#20 0x00000000005c6821 in dispatch_command (command=COM_STMT_EXECUTE, thd=0x2aca1c0ef030, packet=0x2aca1c0f21b1 "\a",
    packet_length=241183368) at sql_parse.cc:1225
#21 0x00000000005c79f0 in do_command (thd=0x2aca1c0ef030) at sql_parse.cc:895
#22 0x00000000005b9af2 in handle_one_connection (arg=0x2aca1c0ef030) at sql_connect.cc:1738
#23 0x0000003d2760673d in start_thread () from /lib64/libpthread.so.0
#24 0x0000003d26ed3d1d in clone () from /lib64/libc.so.6

[Vadim Tkachenko (vadim-tk)]

stack trace for all threads

http://pastebin.com/gftNhxrQ

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Strange,
The adaptive hash index is empty when restarted even reusing shm.
The buffer pool shm code doesn't touch the adaptive hash index.
I continue to analyze.

[Yasufumi Kinoshita (yasufumi-kinoshita)]

not enough information.
Core file analysis file is needed.

[Vadim Tkachenko (vadim-tk)]

Sometimes it is not crash, but just hang.

You can see stack trace
http://percona.pastebin.com/JYJcLkYR

[Yasufumi Kinoshita (yasufumi-kinoshita)]

It seems to be introduced the bug at
------------------------------------------------------------
revno: 116 [merge]
committer: kinoyasu <kinoyasu@gauntlet4>
branch nick: release-5.1.50-12
timestamp: Mon 2010-09-27 12:58:39 +0900
message:
  fix innodb_buffer_pool_shm.patch for -Wpointer-arith warnings of GCC
------------------------------------------------------------
I will fix soon.

[Yasufumi Kinoshita (yasufumi-kinoshita)]

The fix seems not to be enough....

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-430