crash from select from IS.INNODB_SYS_TABLES

[Mark Callaghan]

select from IS.INNODB_SYS_TABLES crashes the server using Percona-Server-5.1.54-rel12.5

Why are there are no tests in mysql-test for this feature?

1) add this to the end of mysql-test/suite/innodb_plugin/t/innodb.test
 select * from INNODB_SYS_TABLES;
2) run that test
3) segfault

The crash happens on the next iteration of the for loop in i_s_innodb_schema_table_fill after executing this code:
                if (rec_get_deleted_flag(rec, 0)) {
                        /* record marked as deleted */
                        btr_pcur_close(&pcur);
                        mtr_commit(&mtr);
                        continue;
                }

The crash stack:

#0 0x00007f3fa9539cfc in pthread_kill () from /lib/libpthread.so.0
#0 0x00007f3fa9539cfc in pthread_kill () from /lib/libpthread.so.0
#1 0x0000000000a2153e in my_write_core (sig=11) at stacktrace.c:333
#2 0x0000000000604b6b in handle_segfault (sig=11) at mysqld.cc:2652
#3 <signal handler called>
#4 0x000000000087f805 in mach_read_from_2 (b=0x2a <Address 0x2a out of bounds>) at ./include/mach0data.ic:84
#5 0x000000000087ff84 in page_header_get_field (page=0x0, field=4) at ./include/page0page.ic:119
#6 0x000000000087ffa3 in page_is_comp (page=0x0) at ./include/page0page.ic:238
#7 0x000000000087ffca in page_rec_is_comp (rec=0x0) at ./include/page0page.ic:250
#8 0x00000000008801f7 in page_rec_get_next (rec=0x0) at ./include/page0page.ic:739
#9 0x00000000008802ca in page_cur_move_to_next (cur=0x7f3f96c2e248) at ./include/page0cur.ic:185
#10 0x0000000000880494 in btr_pcur_move_to_next_on_page (cursor=0x7f3f96c2e240) at ./include/btr0pcur.ic:288
#11 0x0000000000880522 in btr_pcur_move_to_next_user_rec (cursor=0x7f3f96c2e240, mtr=0x7f3f96c2e370) at ./include/btr0pcur.ic:352
#12 0x00000000008862e9 in i_s_innodb_schema_table_fill (thd=0x27c18d0, tables=0x1b73300, cond=0x0) at handler/i_s.cc:3629
#13 0x00000000007c3924 in get_schema_tables_result (join=0x27c45c0, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6715
#14 0x000000000068e5a2 in JOIN::exec (this=0x27c45c0) at sql_select.cc:1878
#15 0x0000000000690ce0 in mysql_select (thd=0x27c18d0, rref_pointer_array=0x27c3c08, tables=0x1b73300, wild_num=1, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2684635648, result=0x28009e0, unit=0x27c35d8, select_lex=0x27c3a10) at sql_select.cc:2565
#16 0x00000000006889fc in handle_select (thd=0x27c18d0, lex=0x27c3538, result=0x28009e0, setup_tables_done_option=0) at sql_select.cc:284
#17 0x0000000000621977 in execute_sqlcom_select (thd=0x27c18d0, all_tables=0x1b73300) at sql_parse.cc:5232
#18 0x0000000000618780 in mysql_execute_command (thd=0x27c18d0) at sql_parse.cc:2371
#19 0x00000000006240e7 in mysql_parse (thd=0x27c18d0, rawbuf=0x1b72cb0 "select * from INNODB_SYS_TABLES", length=31, found_semicolon=0x7f3f96c308b0) at sql_parse.cc:6210
#20 0x0000000000615d8e in dispatch_command (command=COM_QUERY, thd=0x27c18d0, packet=0x27d37d1 "select * from INNODB_SYS_TABLES", packet_length=31) at sql_parse.cc:1278
#21 0x0000000000614cef in do_command (thd=0x27c18d0) at sql_parse.cc:903
#22 0x0000000000612e92 in handle_one_connection (arg=0x27c18d0) at sql_connect.cc:1738
#23 0x00007f3fa9534971 in start_thread () from /lib/libpthread.so.0
#24 0x00007f3fa807a92d in clone () from /lib/libc.so.6
#25 0x0000000000000000 in ?? ()

[Mark Callaghan (mdcallag)]

This diff fixes the crash for me. I don't know much about this part of InnoDB but I looked at other code that uses a cursor and I think it is wrong to close the cursor and commit the minitransaction on the first deleted row, and then continue to use the cursor.

2505c2505,2508
< btr_pcur_move_to_next_user_rec(&pcur, &mtr);
---
> if (!btr_pcur_move_to_next_user_rec(&pcur, &mtr)) {
> mtr_commit(&mtr);
> break;
> }
2516,2517d2518
< btr_pcur_close(&pcur);
< mtr_commit(&mtr);

[Mark Callaghan (mdcallag)]

I think the loop can be simplified as there is no need to call btr_pcur_is_on_user_rec if you check for TRUE returned from btr_pcur_move_to_next_user_rec

        for (;;) {
                if (!btr_pcur_move_to_next_user_rec(&pcur, &mtr)) {
                        /* end of index */
                        btr_pcur_close(&pcur);
                        mtr_commit(&mtr);
                        break;
                }

                rec = btr_pcur_get_rec(&pcur);

                if (rec_get_deleted_flag(rec, 0)) {
                        /* record marked as deleted */
                        continue;
                }

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Thank you.

I knew the something wrong with the innodb_show_sys_tables.patch which was made by another.
But it was added to XtraDB forcibly against my anxiety at the time.
Also I don't know whether your fix is enough or not, for now.

I think, ideally, the code about the i_s should be backported from the InnoDB at official MySQL-5.5.6.
Because it seems much more stable from the innodb_show_sys_tables.patch for 5.1.

[Mark Callaghan (mdcallag)]

Wow, just looked at mysql-trunk. I will backport from there.

I don't trust my fix either as I never use code in this area.

On Tue, Feb 1, 2011 at 7:02 PM, Yasufumi Kinoshita
<email address hidden> wrote:
> Thank you.
>
> I knew the something wrong with the innodb_show_sys_tables.patch which was made by another.
> But it was added to XtraDB forcibly against my anxiety at the time.
> Also I don't know whether your fix is enough or not, for now.
>
> I think, ideally, the code about the i_s should be backported from the InnoDB at official MySQL-5.5.6.
> Because it seems much more stable from the innodb_show_sys_tables.patch for 5.1.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/711532
>
> Title:
>  crash from select from IS.INNODB_SYS_TABLES
>
> Status in Percona Server with XtraDB:
>  New
>
> Bug description:
>  select from IS.INNODB_SYS_TABLES crashes the server using Percona-
>  Server-5.1.54-rel12.5
>
>  Why are there are no tests in mysql-test for this feature?
>
>  1) add this to the end of mysql-test/suite/innodb_plugin/t/innodb.test
>   select * from INNODB_SYS_TABLES;
>  2) run that test
>  3) segfault
>
>  The crash happens on the next iteration of the for loop in i_s_innodb_schema_table_fill after executing this code:
>                  if (rec_get_deleted_flag(rec, 0)) {
>                          /* record marked as deleted */
>                          btr_pcur_close(&pcur);
>                          mtr_commit(&mtr);
>                          continue;
>                  }
>
>  The crash stack:
>
>  #0  0x00007f3fa9539cfc in pthread_kill () from /lib/libpthread.so.0
>  #0  0x00007f3fa9539cfc in pthread_kill () from /lib/libpthread.so.0
>  #1  0x0000000000a2153e in my_write_core (sig=11) at stacktrace.c:333
>  #2  0x0000000000604b6b in handle_segfault (sig=11) at mysqld.cc:2652
>  #3  <signal handler called>
>  #4  0x000000000087f805 in mach_read_from_2 (b=0x2a <Address 0x2a out of bounds>) at ./include/mach0data.ic:84
>  #5  0x000000000087ff84 in page_header_get_field (page=0x0, field=4) at ./include/page0page.ic:119
>  #6  0x000000000087ffa3 in page_is_comp (page=0x0) at ./include/page0page.ic:238
>  #7  0x000000000087ffca in page_rec_is_comp (rec=0x0) at ./include/page0page.ic:250
>  #8  0x00000000008801f7 in page_rec_get_next (rec=0x0) at ./include/page0page.ic:739
>  #9  0x00000000008802ca in page_cur_move_to_next (cur=0x7f3f96c2e248) at ./include/page0cur.ic:185
>  #10 0x0000000000880494 in btr_pcur_move_to_next_on_page (cursor=0x7f3f96c2e240) at ./include/btr0pcur.ic:288
>  #11 0x0000000000880522 in btr_pcur_move_to_next_user_rec (cursor=0x7f3f96c2e240, mtr=0x7f3f96c2e370) at ./include/btr0pcur.ic:352
>  #12 0x00000000008862e9 in i_s_innodb_schema_table_fill (thd=0x27c18d0, tables=0x1b73300, cond=0x0) at handler/i_s.cc:3629
>  #13 0x00000000007c3924 in get_schema_tables_result (join=0x27c45c0, executed_place=PROCESSED_BY_JOIN_EXEC) at sql_show.cc:6715
>  #14 0x000000000068e5a2 in JOIN::exec (this=0x27c45c0) at sql_select.cc:1878
>  #15 0x0000000000690ce0 in mysql_select (thd=0x27c18d0, rref_pointer_array=0x27c3c08, tables=0x1b73300...

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Note:
This problem is only for 5.1 base version.
Because, for 5.5 base version, 5.5.6 code of the i_s is revived as the same name of the patch at 5.1.
XtraDB on 5.1 and 5.5 use different code currently.

[Valentine Gostev (longbow)]

From 5.1.56:

innodb_plugin.innodb [ fail ]
        Test ended at 2011-05-15 19:48:59

CURRENT_TEST: innodb_plugin.innodb
mysqltest: At line 2576: query 'select * from information_schema.INNODB_SYS_TABLES' failed: 2013: Lost connection to MySQL server during query

The result from queries just before the failure was:
< snip >
CREATE TABLE t1 (c1 int default NULL,
c2 int default NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
TRUNCATE TABLE t1;
affected rows: 0
INSERT INTO t1 VALUES (1, 1), (2, 2), (3, 3), (4, 4), (5, 5);
affected rows: 5
info: Records: 5 Duplicates: 0 Warnings: 0
TRUNCATE TABLE t1;
affected rows: 0
DROP TABLE t1;
Variable_name Value
Handler_update 0
Variable_name Value
Handler_delete 0
Variable_name Value
Handler_update 1
Variable_name Value
Handler_delete 1
DROP TABLE bug58912;

More results from queries before failure can be found in /usr/share/mysql-test/var/log/innodb.log

Server [mysqld.1 - pid: 3654, winpid: 3654, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
110515 18:48:34 [ERROR] Cannot find index v_2 in InnoDB index translation table.
110515 18:48:41 InnoDB: Error: in ALTER TABLE `test`.`t1`
InnoDB: has or is referenced in foreign key constraints
InnoDB: which are not compatible with the new table definition.
110515 18:48:41 [Warning] Invalid (old?) table or database name '#sql-e47_6'
110515 18:48:41 InnoDB: Error: table `test`.`t1` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting.html
110515 18:48:42 InnoDB: Error: in RENAME TABLE table `test`.`t1`
InnoDB: is referenced in foreign key constraints
InnoDB: which are not compatible with the new table definition.
110515 18:48:57 InnoDB: Error: in ALTER TABLE `test`.`t2`
InnoDB: has or is referenced in foreign key constraints
InnoDB: which are not compatible with the new table definition.
110515 18:48:57 [Warning] Invalid (old?) table or database name '#sql-e47_6'
110515 18:48:57 InnoDB: Error: table `test`.`t2` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/innodb-troubleshooting.html
110515 18:48:59 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=11
m...

[Stewart Smith (stewart)]

Valentine, it looks as though you've verified this bug? If so, will remove you from being assigned to it and get someone to work on fix.

[Vladislav Vaintroub (wlad-montyprogram)]

reproduced with maria

Thread pointer: 0xc206080
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
015D1AD6 mysqld.exe!mach_read_from_2()[mach0data.ic:84]
01560C83 mysqld.exe!page_header_get_field()[page0page.ic:119]
01560D7E mysqld.exe!page_is_comp()[page0page.ic:238]
01560DA2 mysqld.exe!page_rec_is_comp()[page0page.ic:250]
0156130D mysqld.exe!page_rec_get_next()[page0page.ic:739]
015D9ECF mysqld.exe!page_cur_move_to_next()[page0cur.ic:185]
015DC530 mysqld.exe!btr_pcur_move_to_next_on_page()[btr0pcur.ic:288]
015DD43F mysqld.exe!btr_pcur_move_to_next_user_rec()[btr0pcur.ic:352]
015BC514 mysqld.exe!i_s_innodb_schema_table_fill()[i_s.cc:4340]
0132A5B7 mysqld.exe!get_schema_tables_result()[sql_show.cc:6678]
01400624 mysqld.exe!JOIN::exec()[sql_select.cc:1858]
0140157C mysqld.exe!mysql_select()[sql_select.cc:2567]
014018AA mysqld.exe!handle_select()[sql_select.cc:280]
012FE626 mysqld.exe!execute_sqlcom_select()[sql_parse.cc:5139]
01300BF9 mysqld.exe!mysql_execute_command()[sql_parse.cc:2285]
01305545 mysqld.exe!mysql_parse()[sql_parse.cc:6142]
01305EFC mysqld.exe!dispatch_command()[sql_parse.cc:1215]
01306A6B mysqld.exe!do_command()[sql_parse.cc:907]
0132D0FF mysqld.exe!handle_one_connection()[sql_connect.cc:1177]
01624FBD mysqld.exe!pthread_start()[my_winthread.c:87]
015F8511 mysqld.exe!_callthreadstart()[thread.c:259]
015F858F mysqld.exe!_threadstart()[thread.c:241]
75B733CA kernel32.dll!BaseThreadInitThunk()
77499ED2 ntdll.dll!RtlInitializeExceptionChain()
77499EA5 ntdll.dll!RtlInitializeExceptionChain()

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0C2574F0): =select * from information_schema.innodb_sys_tables
Connection ID (thread ID): 2
Status: NOT_KILLED