server crashed even set innodb_corrupt_table_action=salvage

[zhai weixiang]

There is a corrupted page in ibd. It's a leaf page.
use percona server 5.6.22
set innodb_corrupt_table_action=salvage

(gdb) bt
#0 page_rec_get_n_recs_before (rec=0x0) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/page/page0page.cc:1632
#1 0x0000000000ab16b3 in btr_cur_add_path_info (index=0x2ae6ad4335e8, level=0, tuple=0x2ae6aa2b4c68, mode=1, latch_mode=1, cursor=0x2ae6af70b040, has_search_latch=0,
    file=0xd3a738 "/u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc", line=3872, mtr=0x2ae6af70aba0)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc:3672
#2 btr_cur_search_to_nth_level (index=0x2ae6ad4335e8, level=0, tuple=0x2ae6aa2b4c68, mode=1, latch_mode=1, cursor=0x2ae6af70b040, has_search_latch=0,
    file=0xd3a738 "/u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc", line=3872, mtr=0x2ae6af70aba0)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc:814
#3 0x0000000000ab2224 in btr_estimate_n_rows_in_range (index=0x2ae6ad4335e8, tuple1=<value optimized out>, mode1=<value optimized out>, tuple2=0x2ae6aa2b4cb0, mode2=3)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc:3872
#4 0x00000000009b8518 in ha_innobase::records_in_range (this=0x2ae6a0b5b110, keynr=<value optimized out>, min_key=0x2ae6af70b1b0, max_key=0x2ae6af70b1d0)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/handler/ha_innodb.cc:11217
#5 0x00000000005936f0 in handler::multi_range_read_info_const (this=0x2ae6a0b5b110, keyno=0, seq=0x2ae6af70b570, seq_init_param=<value optimized out>, n_ranges_arg=<value optimized out>,
    bufsz=<value optimized out>, flags=0x2ae6af70b25c, cost=0x2ae6af70b690) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/handler.cc:5980
#6 0x00000000005993c3 in DsMrr_impl::dsmrr_info_const (this=0x2ae6a0b5b5a8, keyno=0, seq=<value optimized out>, seq_init_param=<value optimized out>, n_ranges=<value optimized out>,
    bufsz=0x2ae6af70b708, flags=0x2ae6af70b70c, cost=0x2ae6af70b690) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/handler.cc:6622
#7 0x00000000008220d2 in check_quick_select (param=0x2ae6af70b820, idx=<value optimized out>, index_only=<value optimized out>, tree=<value optimized out>, update_tbl_stats=true,
    mrr_flags=<value optimized out>, bufsize=0x2ae6af70b708, cost=0x2ae6af70b690) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/opt_range.cc:9503
#8 0x00000000008241fe in get_key_scans_params (param=0x2ae6af70b820, tree=0x2ae6a1482078, index_read_must_be_used=false, update_tbl_stats=true, read_time=213764.90000000002)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/opt_range.cc:5600
#9 0x000000000082502e in SQL_SELECT::test_quick_select (this=0x2ae6a0bd5638, thd=0x2ae6a0bee000, keys_to_use=..., prev_tables=<value optimized out>, limit=<value optimized out>,
    force_quick_range=<value optimized out>, interesting_order=st_order::ORDER_NOT_RELEVANT) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/opt_range.cc:2900
#10 0x0000000000865784 in get_quick_record_count (join=0x2ae6a141f060, tables_arg=0x2ae6a141e248, conds=0x2ae6a141ee60, keyuse_array=0x2ae6a141f288, first_optimization=255)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_optimizer.cc:4001
#11 make_join_statistics (join=0x2ae6a141f060, tables_arg=0x2ae6a141e248, conds=0x2ae6a141ee60, keyuse_array=0x2ae6a141f288, first_optimization=255)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_optimizer.cc:3712
#12 0x0000000000867211 in JOIN::optimize (this=0x2ae6a141f060) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_optimizer.cc:382
#13 0x000000000070bcdd in mysql_execute_select (thd=0x2ae6a0bee000, tables=0x2ae6a141e248, wild_num=1, fields=<value optimized out>, conds=0x2ae6a141ee60, order=<value optimized out>,
    group=0x2ae6a0bf0d10, having=0x0, select_options=2147748608, result=0x2ae6a141f038, unit=0x2ae6a0bf05c8, select_lex=0x2ae6a0bf0c10)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_select.cc:1086
#14 mysql_select (thd=0x2ae6a0bee000, tables=0x2ae6a141e248, wild_num=1, fields=<value optimized out>, conds=0x2ae6a141ee60, order=<value optimized out>, group=0x2ae6a0bf0d10, having=0x0,
    select_options=2147748608, result=0x2ae6a141f038, unit=0x2ae6a0bf05c8, select_lex=0x2ae6a0bf0c10) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_select.cc:1221
#15 0x000000000070c61f in handle_select (thd=0x2ae6a0bee000, result=0x2ae6a141f038, setup_tables_done_option=0) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_select.cc:110
#16 0x00000000006e4515 in execute_sqlcom_select (thd=0x2ae6a0bee000, all_tables=0x2ae6a141e248) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:5630
#17 0x00000000006e9195 in mysql_execute_command (thd=0x2ae6a0bee000) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:2988
#18 0x00000000006ec0e8 in mysql_parse (thd=0x2ae6a0bee000, rawbuf=<value optimized out>, length=54, parser_state=<value optimized out>)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:6918
#19 0x00000000006ed7d1 in dispatch_command (command=COM_QUERY, thd=0x2ae6a0bee000, packet=0x2ae6af70fab0 "", packet_length=54)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:1434
#20 0x00000000006b55f5 in do_handle_one_connection (thd_arg=<value optimized out>) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_connect.cc:1532
#21 0x00000000006b5772 in handle_one_connection (arg=0x2ae6aa30d000) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_connect.cc:1443
#22 0x000000000093d2c7 in pfs_spawn_thread (arg=0x2ae6897232a0) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/perfschema/pfs.cc:1860
#23 0x000000372ca07851 in start_thread () from /lib64/libpthread.so.0
#24 0x000000372c6e767d in clone () from /lib64/libc.so.6
(gdb) f 0
#0 page_rec_get_n_recs_before (rec=0x0) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/page/page0page.cc:1632
1632 while (rec_get_n_owned_new(rec) == 0) {
(gdb) p rec
$23 = (const rec_t *) 0x0
(gdb) f 2
#2 btr_cur_search_to_nth_level (index=0x2ae6ad4335e8, level=0, tuple=0x2ae6aa2b4c68, mode=1, latch_mode=1, cursor=0x2ae6af70b040, has_search_latch=0,
    file=0xd3a738 "/u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc", line=3872, mtr=0x2ae6af70aba0)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0cur.cc:814
814 tes,
(gdb) p *((buf_page_t *)(cursor->page_cur.block))
$24 = {space = 2898, offset = 6151, buf_fix_count = 1, io_fix = 0 '\000', state = 5 '\005', flush_type = 0, buf_pool_index = 2, zip = {data = 0x0, m_end = 0, m_nonempty = 0, n_blobs = 0,
    ssize = 0}, hash = 0x0, list = {prev = 0x2ae4f7fff400, next = 0x0}, newest_modification = 0, oldest_modification = 0, LRU = {prev = 0x2ae4f7fff400, next = 0x2ae4f7fff700}, old = 0,
  freed_page_clock = 0, access_time = 1903998463, is_corrupt = 1}
(gdb) p srv_pass_corrupt_table
$25 = 2
(gdb)

[zhai weixiang (zhaiwx1987)]

'check table' can also corrupt the server:

for example: check table sbtest1;

(gdb) bt
#0 0x000000372c6328a5 in raise () from /lib64/libc.so.6
#1 0x000000372c634085 in abort () from /lib64/libc.so.6
#2 0x0000000000aa1be4 in btr_page_get_father_node_ptr_func (offsets=<value optimized out>, heap=0x2b6de6c7c300, cursor=<value optimized out>, line=<value optimized out>,
    mtr=<value optimized out>, file=0xd397c8 "/u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0btr.cc")
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0btr.cc:1492
#3 0x0000000000aa3b28 in btr_page_get_father_block (index=0x2b6df24435e8, trx=0x2b6de6c29a68) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0btr.cc:1521
#4 btr_validate_level (index=0x2b6df24435e8, trx=0x2b6de6c29a68) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0btr.cc:4907
#5 btr_validate_index (index=0x2b6df24435e8, trx=0x2b6de6c29a68) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/btr/btr0btr.cc:5050
#6 0x00000000009b595e in ha_innobase::check (this=0x2b6de5b5eb10, thd=0x2b6de5bd7000, check_opt=0x2b6de5bda2d8)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/innobase/handler/ha_innodb.cc:12090
#7 0x000000000059c82a in handler::ha_check (this=0x2b6de5b5eb10, thd=0x2b6de5bd7000, check_opt=0x2b6de5bda2d8) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/handler.cc:4287
#8 0x0000000000839e05 in mysql_admin_table (thd=0x2b6de5bd7000, tables=0x0, check_opt=0x2b6de5bda2d8, operator_name=0xd55ebc "check", lock_type=TL_READ_NO_INSERT, open_for_modify=false,
    repair_table_use_frm=false, extra_open_options=32, prepare_func=0, operator_func=(int (handler::*)(handler *, THD *, HA_CHECK_OPT *)) 0x59c740 <handler::ha_check(THD*, HA_CHECK_OPT*)>,
    view_operator_func=0x75db50 <view_checksum(THD*, TABLE_LIST*)>) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_admin.cc:654
#9 0x000000000083aa5b in Sql_cmd_check_table::execute (this=<value optimized out>, thd=0x2b6de5bd7000) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_admin.cc:1108
#10 0x00000000006e86cb in mysql_execute_command (thd=0x2b6de5bd7000) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:5444
#11 0x00000000006ec0e8 in mysql_parse (thd=0x2b6de5bd7000, rawbuf=<value optimized out>, length=19, parser_state=<value optimized out>)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:6918
#12 0x00000000006ed7d1 in dispatch_command (command=COM_QUERY, thd=0x2b6de5bd7000, packet=0x2b6df470fab0 "", packet_length=19)
    at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_parse.cc:1434
#13 0x00000000006b55f5 in do_handle_one_connection (thd_arg=<value optimized out>) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_connect.cc:1532
#14 0x00000000006b5772 in handle_one_connection (arg=0x2b6bb0f2f000) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/sql/sql_connect.cc:1443
#15 0x000000000093d2c7 in pfs_spawn_thread (arg=0x2b6dce7232a0) at /u01/project/mysql-lab/percona-server-5.6.22-71.0/storage/perfschema...

[zhai weixiang (zhaiwx1987)]

There is another interesting thing :

root@sb1 12:36:04>show variables like 'innodb_corrupt_table_action';
+-----------------------------+---------+
| Variable_name | Value |
+-----------------------------+---------+
| innodb_corrupt_table_action | salvage |
+-----------------------------+---------+
1 row in set (0.01 sec)

root@sb1 12:36:13>select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 997936 |
+----------+
1 row in set (0.64 sec)

from error log:
InnoDB: space 2898 will be treated as corrupt.
InnoDB: Index corruption: rec offs 125 next offs 0, page no 4417,
InnoDB: index `k` of table `sb1`.`sbtest1`. We try to skip the rest of the page.
InnoDB: Index corruption: rec offs 125 next offs 0, page no 8024,
InnoDB: index `k` of table `sb1`.`sbtest1`. We try to skip the rest of the page.

root@sb1 12:37:52>select count(*) from sbtest1 force index(primary);
+----------+
| count(*) |
+----------+
| 999854 |
+----------+
1 row in set (3.73 sec)

from error log:
InnoDB: Index corruption: rec offs 127 next offs 0, page no 6151,
InnoDB: index `PRIMARY` of table `sb1`.`sbtest1`. We try to skip the rest of the page.
InnoDB: Index corruption: rec offs 127 next offs 0, page no 12207,
InnoDB: index `PRIMARY` of table `sb1`.`sbtest1`. We try to skip the rest of the page.

index 'k' is choosed to execute select count(*) though both indexes are corrupted . In this case ,we should use Primary index instead of index 'k'

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3259