InnoDB: Failing assertion: inode && used && mtr in file fsp0fsp.cc line 2188 | handle_fatal_signal (sig=6) in fseg_n_reserved_pages_low

[Roel Van de Paar]

[... (see full error log in bundle)...]
2014-10-06 21:40:01 7faf802bb700InnoDB: unknown error code 47 while dropping table:"test"."t2".
2014-10-06 21:40:01 7faf802bb700 InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
InnoDB: many active transactions running concurrently?
2014-10-06 21:40:01 7faf802bb700 InnoDB: Cannot save table statistics for table "test"."t1": Too many concurrent transactions
2014-10-06 21:40:01 7faf802bb700 InnoDB: Assertion failure in thread 140391746352896 in file fsp0fsp.cc line 2188
InnoDB: Failing assertion: inode && used && mtr

Thread 1 (Thread 0x7faf802bb700 (LWP 27023)):
+bt
#0 0x00007faf9fa8f771 in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000000ab6f6a in my_write_core (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/mysys/stacktrace.c:422
#2 0x000000000072f89f in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x00007faf9e6935c9 in raise () from /lib64/libc.so.6
#5 0x00007faf9e694cd8 in abort () from /lib64/libc.so.6
#6 0x0000000000d37129 in fseg_n_reserved_pages_low (inode=0x0, used=0x7faf802b71a0, mtr=0x7faf802b7200) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/fsp/fsp0fsp.cc:2188
#7 0x0000000000d37324 in fseg_n_reserved_pages (header=0x7faf7b250054 "", used=0x7faf802b71a0, mtr=0x7faf802b7200) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/fsp/fsp0fsp.cc:2230
#8 0x0000000000c83497 in btr_get_size (index=0x7faf40a458c8, flag=2, mtr=0x7faf802b7200) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/btr/btr0btr.cc:1245
#9 0x0000000000d14223 in dict_stats_analyze_index (index=0x7faf40a458c8) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/dict/dict0stats.cc:1910
#10 0x0000000000d14cba in dict_stats_update_persistent (table=0x7faf40b0a568) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/dict/dict0stats.cc:2209
#11 0x0000000000d16871 in dict_stats_update (table=0x7faf40b0a568, stats_upd_option=DICT_STATS_RECALC_PERSISTENT) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/dict/dict0stats.cc:3127
#12 0x0000000000d16b57 in dict_stats_update (table=0x7faf40b0a568, stats_upd_option=DICT_STATS_FETCH_ONLY_IF_NOT_IN_MEMORY) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/dict/dict0stats.cc:3248
#13 0x0000000000accd2d in dict_stats_init (table=0x7faf40b0a568) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/include/dict0stats.ic:174
#14 0x0000000000ad324c in ha_innobase::open (this=0x7faf40ae85b0, name=0x7faf409ea9c8 "./test/t2", mode=2, test_if_locked=2) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/innobase/handler/ha_innodb.cc:5518
#15 0x0000000000641ce5 in handler::ha_open (this=0x7faf40ae85b0, table_arg=0x7faf40a65d00, name=0x7faf409ea9c8 "./test/t2", mode=2, test_if_locked=2) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/handler.cc:2683
#16 0x00000000008ab3eb in open_table_from_share (thd=0x2acb9e0, share=0x7faf409ea690, alias=0x7faf40005188 "t2", db_stat=39, prgflag=44, ha_open_flags=0, outparam=0x7faf40a65d00, is_create_table=false) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/table.cc:2364
#17 0x0000000000770f89 in open_table (thd=0x2acb9e0, table_list=0x7faf40005190, ot_ctx=0x7faf802b8a40) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_base.cc:3205
#18 0x000000000077373e in open_and_process_table (thd=0x2acb9e0, lex=0x2acdf28, tables=0x7faf40005190, counter=0x2acdfe8, flags=0, prelocking_strategy=0x7faf802b8b40, has_prelocking_list=false, ot_ctx=0x7faf802b8a40) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_base.cc:4699
#19 0x0000000000774643 in open_tables (thd=0x2acb9e0, start=0x7faf802b8b00, counter=0x2acdfe8, flags=0, prelocking_strategy=0x7faf802b8b40) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_base.cc:5213
#20 0x00000000007757d2 in open_normal_and_derived_tables (thd=0x2acb9e0, tables=0x7faf40005190, flags=0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_base.cc:5913
#21 0x000000000087a367 in mysql_multi_update_prepare (thd=0x2acb9e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_update.cc:1340
#22 0x00000000007e4caf in mysql_execute_command (thd=0x2acb9e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_parse.cc:3734
#23 0x00000000007ed033 in mysql_parse (thd=0x2acb9e0, rawbuf=0x7faf400050a0 "UPDATE t2,t1 SET t2.a=t1.a+2", length=28, parser_state=0x7faf802b9e30) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_parse.cc:6773
#24 0x00000000007df49c in dispatch_command (command=COM_QUERY, thd=0x2acb9e0, packet=0x32e1671 "UPDATE t2,t1 SET t2.a=t1.a+2;", packet_length=29) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_parse.cc:1432
#25 0x00000000007de3c8 in do_command (thd=0x2acb9e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_parse.cc:1049
#26 0x00000000007a6385 in do_handle_one_connection (thd_arg=0x2acb9e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_connect.cc:1532
#27 0x00000000007a5e8d in handle_one_connection (arg=0x2acb9e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/sql/sql_connect.cc:1443
#28 0x0000000000dc5590 in pfs_spawn_thread (arg=0x3264e20) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-69.0/storage/perfschema/pfs.cc:1860
#29 0x00007faf9fa8adf3 in start_thread () from /lib64/libpthread.so.0
#30 0x00007faf9e75401d in clone () from /lib64/libc.so.6

[Roel Van de Paar (roel11)]

Possibly/likely related to http://bugs.mysql.com/bug.php?id=74343

[Ramesh Sivaraman (rameshvs02)]

**** Testcase

DROP DATABASE test;CREATE DATABASE test;USE test;
CREATE TABLE t(a CHAR (1)) ENGINE=InnoDB;
set global innodb_undo_logs=-1;
set global innodb_trx_rseg_n_slots_debug=1;
drop table if exists t,tp,t1_will_crash;
SET GLOBAL innodb_corrupt_table_action='salvage';
INSERT INTO t VALUES(1);
INSERT INTO t VALUES(1);

When we remove innodb_corrupt_table_action from above testcase getting slightly different assertion, which is similar to MS bug #74343 (http://bugs.mysql.com/bug.php?id=74343).

Laurynas, could you please confirm whether it is similar bug or not

** Testcase
DROP DATABASE test;CREATE DATABASE test;USE test;
CREATE TABLE t(a CHAR (1)) ENGINE=InnoDB;
set global innodb_undo_logs=-1;
set global innodb_trx_rseg_n_slots_debug=1;
drop table if exists t,tp,t1_will_crash;
INSERT INTO t VALUES(1);
INSERT INTO t VALUES(1);
** GDB info

#0 0x00007f7a33b63771 in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000000ab7e66 in my_write_core (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/mysys/stacktrace.c:422
#2 0x000000000072fcd3 in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x00007f7a327675c9 in raise () from /lib64/libc.so.6
#5 0x00007f7a32768cd8 in abort () from /lib64/libc.so.6
#6 0x0000000000d375b4 in fseg_inode_get (header=0x7f79c1fec054 "", space=6, zip_size=0, mtr=0x7f798dff65f0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/fsp/fsp0fsp.cc:1899
#7 0x0000000000d38205 in fseg_n_reserved_pages (header=0x7f79c1fec054 "", used=0x7f798dff6590, mtr=0x7f798dff65f0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/fsp/fsp0fsp.cc:2228
#8 0x0000000000c84393 in btr_get_size (index=0x7f7976c6bff8, flag=2, mtr=0x7f798dff65f0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/btr/btr0btr.cc:1245
#9 0x0000000000d1511f in dict_stats_analyze_index (index=0x7f7976c6bff8) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/dict/dict0stats.cc:1910
#10 0x0000000000d15bb6 in dict_stats_update_persistent (table=0x7f7976c82378) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/dict/dict0stats.cc:2209
#11 0x0000000000d1776d in dict_stats_update (table=0x7f7976c82378, stats_upd_option=DICT_STATS_RECALC_PERSISTENT) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/dict/dict0stats.cc:3127
#12 0x0000000000d1b9cb in dict_stats_process_entry_from_recalc_pool () at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.21-70.0/storage/innobase/dict/dict0stats_bg.cc:313
#13 0x0000000000d1baaf in dict_stats_thread (arg=0x0) at /mnt/workspace/percona-server-5.6-bi...

[Laurynas Biveinis (laurynas-biveinis)]

The 1st issue is the same as in upstream 74343. After that one is fixed, the innodb_corrupt_table_action=salvage testcase should be retried.

[Roel Van de Paar (roel11)]

Set as dup of http://bugs.mysql.com/bug.php?id=74343 ftm

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1552