Update of federated table and MyISAM table with one row crashes server

[Valerii Kravchuk]

See testcase.sql attached. Multiple tables UPDATE involving federated table with one row crashes Percona server 5.1.x (5.1.59 and 5.1.68 at least):

[openxs@centos Percona-Server-5.1.68-rel14.5-513.Linux.x86_64]$ bin/mysql --no-defaults --socket=/tmp/mysql.sock -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.1.68rel14.5 Percona Server with XtraDB (GPL), Release rel14.5, Revision 513

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use testcase
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> UPDATE user_dimension_f ud
    -> INNER JOIN first_traded_data ftd
    -> ON ftd.account_id = ud.account_id
    -> SET ud.first_traded = ftd.first_traded
    -> WHERE ud.first_traded IS NULL;
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 130521 19:56:32 mysqld_safe Number of processes running now: 0
130521 19:56:32 mysqld_safe mysqld restarted

mysql> exit
Bye
[openxs@centos Percona-Server-5.1.68-rel14.5-513.Linux.x86_64]$ tail -60 data/centos.err
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=2
max_threads=151
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338436 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x1a99a20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6fb4562e68 thread_stack 0x40000
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(my_print_stacktrace+0x35)[0x8adb65]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(handle_fatal_signal+0x378)[0x6af438]
/lib64/libpthread.so.0(+0xf500)[0x7f6fb79d4500]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_ZN12ha_federated7rnd_posEPhS0_+0x3f)[0x735b5f]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_ZN12multi_update10do_updatesEv+0x3c9)[0x652e39]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_ZN12multi_update8send_eofEv+0x218)[0x653218]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld[0x62fe15]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_ZN4JOIN4execEv+0x913)[0x63b4f3]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x199)[0x63d0b9]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lex+0x115)[0x6509b5]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1553)[0x5cc463]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x51b)[0x5d15eb]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x46e)[0x5d1a5e]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(_Z10do_commandP3THD+0x128)[0x5d2cf8]
/home/openxs/Percona-Server-5.1.68-rel14.5-513.Linux.x86_64/libexec/mysqld(handle_one_connection+0x7d8)[0x5c4928]
/lib64/libpthread.so.0(+0x7851)[0x7f6fb79cc851]
/lib64/libc.so.6(clone+0x6d)[0x7f6fb6a2411d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f6f94004a00): UPDATE user_dimension_f ud INNER JOIN first_traded_data ftd ON ftd.account_id = ud.account_id SET ud.first_traded = ftd.first_traded WHERE ud.first_traded IS NULL
Connection ID (thread ID): 7
Status: NOT_KILLED

[Valerii Kravchuk (valerii-kravchuk)]

This upstream bug can be related: http://bugs.mysql.com/bug.php?id=68354

[Laurynas Biveinis (laurynas-biveinis)]

Valerii -

Are 5.5 and 5.6 affected? Do you have a GDB-resolved crash stacktrace?

The upstream crash stacktrace looks identical in function names, thus marking that bug as duplicate for now.

[Valerii Kravchuk (valerii-kravchuk)]

Test case from upstream http://bugs.mysql.com/bug.php?id=68354 crashed PS 5.5.30-30.2 also:

stack_bottom = 7fd02c065da8 thread_stack 0x40000
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x7b1b75]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x4b4)[0x68d494]
/lib64/libpthread.so.0[0x3ffac0eeb0]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_ZN12ha_federated7rnd_posEPhS0_+0x35)[0x8fdb65]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_ZN12multi_update10do_updatesEv+0x3ea)[0x5fd7fa]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_ZN12multi_update8send_eofEv+0x2a8)[0x5fdcd8]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld[0x5b45de]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_ZN4JOIN4execEv+0xca0)[0x5c98c0]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5cb0cc]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lexPP12multi_update+0x124)[0x5fc6e4]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0x317a)[0x58e5ba]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x33b)[0x59019b]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15cd)[0x59180d]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x13f)[0x62c33f]
/tmp/Percona-Server-5.5.30-rel30.2-500.Linux.x86_64/bin/mysqld(handle_one_connection+0x51)[0x62c421]
/lib64/libpthread.so.0[0x3ffac06ccb]
/lib64/libc.so.6(clone+0x6d)[0x3ffa8e0c2d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fd014004ba0): UPDATE t1 JOIN t1fed ON t1.id = t1fed.id SET t1fed.message = t1.message

[Valerii Kravchuk (valerii-kravchuk)]

If local table with one row is InnoDB one I see no crash.

[Valerii Kravchuk (valerii-kravchuk)]

PS 5.6.10 is also affected:

stack_bottom = 7f689e040e28 thread_stack 0x40000
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x8da155]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x3fb)[0x65788b]
/lib64/libpthread.so.0[0x3ffac0eeb0]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_ZN12ha_federated7rnd_posEPhS0_+0x3f)[0x8e893f]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x6e)[0x586d9e]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_ZN12multi_update10do_updatesEv+0x33a)[0x733e4a]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_ZN12multi_update8send_eofEv+0x298)[0x735148]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_ZN4JOIN4execEv+0x3da)[0x6ae41a]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld[0x6f5589]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xbc)[0x6f56bc]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lexPP12multi_update+0x144)[0x7352c4]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0x4cf6)[0x6d48e6]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x318)[0x6d6768]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x153e)[0x6d7d9e]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z10do_commandP3THD+0xd5)[0x6d8415]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x11f)[0x6a2ddf]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(handle_one_connection+0x45)[0x6a2ec5]
/tmp/Percona-Server-5.6.10-alpha60.2-324.Linux.x86_64/bin/mysqld(pfs_spawn_thread+0x13b)[0xaf6e6b]
/lib64/libpthread.so.0[0x3ffac06ccb]
/lib64/libc.so.6(clone+0x6d)[0x3ffa8e0c2d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f6884004f00): UPDATE user_dimension_f ud INNER JOIN first_traded_data ftd ON ftd.account_id = ud.account_id SET ud.first_traded = ftd.first_traded WHERE ud.first_traded IS NULL
Connection ID (thread ID): 1
Status: NOT_KILLED

[Laurynas Biveinis (laurynas-biveinis)]

Upstream claims to have fixed this in 5.5.36 and 5.6.16:

5.5$ bzr log -r 4569
------------------------------------------------------------
revno: 4569
committer: Arun Kuruvila <email address hidden>
branch nick: mysql-5.5
timestamp: Mon 2013-12-30 11:39:55 +0530
message:
  Bug #16324629 : SERVER CRASHES ON UPDATE/JOIN FEDERATED +
                  LOCAL TABLE WHEN ONLY 1 LOCAL ROW

  Description: When updating a federated table with UPDATE...
  JOIN, the server consistently crashes with Signal 11 when
  only 1 row exists in the local table involved in the join
  and that 1 row can be joined with a row in the federated
  table.

  Analysis: Interaction between the federated engine and the
  optimizer results in the crash. In our scenario, ie, local
  table having only one row, the program is following a
  different path because the table is treated as a constant
  table by the join optimizer. So in this scenario
  "index_read()" is happening in the prepare phase,
  since optimizer plan is different for constant table joins.
  In this case, "index_read_idx_map()" (inside handler.cc) is
  calling "index_read()" and inside "index_read()", matching
  rows are fetched and "stored_result" gets populated by
  calling "store_result()". And just after "index_read()",
  "index_end()" function is called. And in the "index_end()",
  its freeing the "stored_result" by calling "free_result()".
  So when it reaches the execution phase, in "position()"
  function, we are getting assertion at
  "DBUG_ASSERT(stored_result);". In all other scenarios (ie,
  table with more than 1 row), optimizer plan is different
  and "index_read()" is happening in the execution phase.

  Fix: So my fix is to have a separate ha_federated member
  function for "index_read_idx_map()" which will handle
  federated engine separately. So that position() will be
  called before index_end() call in constant table scenario.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-96