Crash on wsrep_append_foreign_key

[Jervin R]

Customer was running 5.6.19-25.6 RPMs and resolved this dump from a crash. Could not find any similar bug reports, there are no core file unfortunately.

0x6635c4 handle_fatal_signal + 1220
0x7f5f59927710 _end + 1482463336
0x7f5f57d52925 _end + 1453279869
0x7f5f57d54105 _end + 1453285981
0x93e75d std::_Rb_tree<unsigned long, std::pair<unsigned long const, page_zip_stat_t>, std::_Select1st<std::pair<unsigned long const, pa + 57533
0x943cb7 std::_Rb_tree<unsigned long, std::pair<unsigned long const, page_zip_stat_t>, std::_Select1st<std::pair<unsigned long const, pa + 79383
0x944b04 std::_Rb_tree<unsigned long, std::pair<unsigned long const, page_zip_stat_t>, std::_Select1st<std::pair<unsigned long const, pa + 83044
0x944ef9 std::_Rb_tree<unsigned long, std::pair<unsigned long const, page_zip_stat_t>, std::_Select1st<std::pair<unsigned long const, pa + 84057
0x945243 std::_Rb_tree<unsigned long, std::pair<unsigned long const, page_zip_stat_t>, std::_Select1st<std::pair<unsigned long const, pa + 84899
0x9b6b1f std::_Rb_tree<unsigned long, std::pair<unsigned long const, row_log_table_blob_t>, std::_Select1st<std::pair<unsigned long cons + 17615
0x9bccf4 std::_Rb_tree<unsigned long, std::pair<unsigned long const, row_log_table_blob_t>, std::_Select1st<std::pair<unsigned long cons + 42660
0x9104a9 wsrep_append_foreign_key(trx_t*, dict_foreign_t*, unsigned char const*, dict_index_t*, unsigned long, unsigned long) + 16089
0x59ef10 handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function) + 144
0x5a1517 handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool) + 103
0x5a0095 handler::multi_range_read_next(char**) + 133
0x5a1a58 DsMrr_impl::dsmrr_next(char**) + 296
0x807ed2 QUICK_RANGE_SELECT::get_next() + 82
0x82903b init_read_record_idx(READ_RECORD*, THD*, TABLE*, bool, unsigned int, bool) + 1099
0x75c23c mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, enum_duplicates, + 4444
0x6e8155 mysql_execute_command(THD*) + 4213
0x6ed768 mysql_parse(THD*, char*, unsigned int, Parser_state*) + 1624
0x6ed8c1 mysql_parse(THD*, char*, unsigned int, Parser_state*) + 1969
0x6ef412 dispatch_command(enum_server_command, THD*, char*, unsigned int) + 4962
0x6f0f73 do_command(THD*) + 483
0x6b9bdf do_handle_one_connection(THD*) + 383
0x6b9db7 handle_one_connection + 71
0xae9b4a pfs_spawn_thread + 298

[Seppo Jaakola (seppo-jaakola)]

This crash happens while processing an UPDATE statement. Can you find out if there was simultaneous ALTER TABLE processing on the same table?

[Jervin R (revin)]

@Seppo,

Unfortunately, the customer cannot tell if there was an ALTER, I guess the best path is to update to 5.6.20 and see if this happens again without a test case for now.

[Jervin R (revin)]

@Seppo,

Here is another crash which clearly shows an ALTER, we do not have the ALTER statement nor the table though, this is the same PXC 5.6.20 version different server.

[root@t2-mysql02 percona]# resolve_stack_dump -s mysqld.symbols -n t|c++filt
0x8f5835 my_print_stacktrace + 53
0x664384 handle_fatal_signal + 1204
0x3a0d20f710 _end + 199934632
0x3a0cb282aa _end + 192695874
0xa5ba31 std::_Rb_tree<dict_foreign_t*, dict_foreign_t*, std::_Identity<dict_foreign_t*>, dict_foreign_compare, std::allocator<dict_foreign_t*> >::_M_insert_unique(dict_foreign_t* const&) + 81
0xa59307 wsrep_dict_foreign_find_index(dict_table_t*, char const**, char const**, unsigned long, dict_index_t*, unsigned long, unsigned long) + 34727
0xa5b5a5 wsrep_dict_foreign_find_index(dict_table_t*, char const**, char const**, unsigned long, dict_index_t*, unsigned long, unsigned long) + 43589
0x9af6c1 PageConverter::~PageConverter() + 100401
0x913a09 wsrep_append_foreign_key(trx_t*, dict_foreign_t*, unsigned char const*, dict_index_t*, unsigned long, unsigned long) + 18953
0x5a7ab1 ha_create_table(THD*, char const*, char const*, char const*, st_ha_create_information*, bool, bool) + 561
0x744ce0 mysql_alter_table(THD*, char*, char*, st_ha_create_information*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool) + 9376
0x835487 Sql_cmd_alter_table::execute(THD*) + 1351
0x6e990e mysql_execute_command(THD*) + 5006
0x6eedc8 mysql_parse(THD*, char*, unsigned int, Parser_state*) + 1624
0x893925 Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned int) + 1909
0x8976b4 Log_event::apply_event(Relay_log_info*) + 116
0x59c0e6 wsrep_apply_cb(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*) + 1654
0x7f42b8390a81 _end + -1224550887
0x7f42b83c8965 _end + -1224321795
0x7f42b83c9383 _end + -1224319205
0x7f42b83cdd35 _end + -1224300339
0x7f42b83a76ca _end + -1224457630
0x7f42b83a7ce3 _end + -1224456069
0x7f42b83c5ef3 _end + -1224332661
0x7f42b83dcba3 _end + -1224239301
0x59cf31 wsrep_client_rollback(THD*) + 881
0x58866e start_wsrep_THD + 878
0x3a0d2079d1 _end + 199902569
0x3a0cae886d _end + 192435205
[root@t2-mysql02 percona]#

[Jervin R (revin)]

We may have a test case for this on i45821 but have to narrow it down based on the SQL file as it has a number of ALTER statements.

@Seppo, let us know if you would like to see that file.

[Seppo Jaakola (seppo-jaakola)]

@Jervin, sure SQL file would be good to look at, you can send e.g. to my private email

[Frederic Descamps (lefred)]

I've a core dump and I resolved it where can I send it ?

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Jervin, @Fred,

Is it still possible to reproduce this with 5.6.21 PXC?

[Nilnandan Joshi (nilnandan-joshi)]

Set to "Incomplete" as per last comment.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1724