SHOW STATUS in parallel to online buffer pool resizing may crash

[Laurynas Biveinis]

2016-04-28T02:11:55.318656Z 0 [Note] InnoDB: Resizing also other hash tables.
02:11:55 UTC - mysqld got signal 11 ;
...
Thread 1 (Thread 0x8a05ab40 (LWP 3478)):
#0 0xb77b4420 in __kernel_vsyscall ()
#1 0xb7531ff3 in __pthread_kill (threadid=2315627328, signo=signo@entry=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#2 0x08b33c37 in my_write_core (sig=11) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/mysys/stacktrace.c:247
#3 0x083a43b1 in handle_fatal_signal (sig=11) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/signal_handler.cc:223
#4 <signal handler called>
#5 srv_export_innodb_status () at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/storage/innobase/srv/srv0srv.cc:1574
#6 0x08b5bc01 in innodb_export_status () at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/storage/innobase/handler/ha_innodb.cc:17266
#7 show_innodb_vars (thd=0xa16a228, var=0x8a057598, buff=0x8a057608 "\006") at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/storage/innobase/handler/ha_innodb.cc:20361
#8 0x0893f936 in show_status_array (thd=thd@entry=0xa16a228, wild=wild@entry=0x0, variables=0x9b00210, value_type=OPT_GLOBAL, status_var=status_var@entry=0x8a057a8c, prefix=prefix@entry=0x91bd8db "", tl=tl@entry=0x9f3ef40, ucase_names=true, cond=cond@entry=0x9f3f4f8) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_show.cc:3000
#9 0x0894788c in fill_status (thd=0xa16a228, tables=0x9f3ef40, cond=0x9f3f4f8) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_show.cc:7914
#10 0x0893192f in do_fill_table (thd=thd@entry=0xa16a228, table_list=table_list@entry=0x9f3ef40, qep_tab=qep_tab@entry=0xa1b4dc4) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_show.cc:8621
#11 0x0894680c in get_schema_tables_result (join=join@entry=0xa1b4a80, executed_place=executed_place@entry=PROCESSED_BY_JOIN_EXEC) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_show.cc:8723
#12 0x089272e8 in JOIN::prepare_result (this=this@entry=0xa1b4a80) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_select.cc:909
#13 0x088aff69 in JOIN::exec (this=0xa1b4a80) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_executor.cc:124
#14 0x08927c55 in handle_query (thd=thd@entry=0xa16a228, lex=lex@entry=0xa16bb40, result=0x9f3f890, added_options=0, removed_options=0) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_select.cc:184
#15 0x08356a2d in execute_sqlcom_select (thd=thd@entry=0xa16a228, all_tables=<optimized out>) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:6024
#16 0x088e7a16 in mysql_execute_command (thd=thd@entry=0xa16a228, first_level=first_level@entry=true) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:3239
#17 0x088e9ba0 in mysql_parse (thd=thd@entry=0xa16a228, parser_state=parser_state@entry=0x8a059608) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:6623
#18 0x088e9c82 in wsrep_mysql_parse (thd=thd@entry=0xa16a228, rawbuf=0x9f3dce0 "SELECT SUBSTR(variable_value, 1, 34) = 'Completed resizing buffer pool at '\n FROM information_schema.global_status\nWHERE LOWER(variable_name) = 'innodb_buffer_pool_resize_status'", length=179, parser_state=parser_state@entry=0x8a059608) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:6318
#19 0x088eb53b in dispatch_command (thd=thd@entry=0xa16a228, com_data=com_data@entry=0x8a059e50, command=COM_QUERY) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:1693
#20 0x088ed4fd in do_command (thd=thd@entry=0xa16a228) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/sql_parse.cc:1129
#21 0x089bf338 in handle_connection (arg=arg@entry=0x9eef480) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/sql/conn_handler/connection_handler_per_thread.cc:313
#22 0x08ef9dd3 in pfs_spawn_thread (arg=0x9ee0bf8) at /mnt/workspace/pxc57.build/BUILD_TYPE/release/label_exp/ubuntu-trusty-32bit/storage/perfschema/pfs.cc:2192
#23 0xb752cf70 in start_thread (arg=0x8a05ab40) at pthread_create.c:312
#24 0xb7317bee in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:129

The crashing line is

  mem_adaptive_hash += mem_heap_get_size(ht->heap);

in srv_export_innodb_status. The surrounding piece of code is

 ut_ad(btr_search_sys->hash_tables);

 for (i = 0; i < btr_ahi_parts; i++) {
  hash_table_t* ht = btr_search_sys->hash_tables[i];

  ut_ad(ht);
  ut_ad(ht->heap);
  /* Multiple mutexes/heaps are currently never used for adaptive
  hash index tables. */
  ut_ad(!ht->n_sync_obj);
  ut_ad(!ht->heaps);

  mem_adaptive_hash += mem_heap_get_size(ht->heap);
  mem_adaptive_hash += ht->n_cells * sizeof(hash_cell_t);
 }

So we read AHI hash tables without any locking. But online buffer pool resize may make those pointers dangling, in parallel:

void
btr_search_sys_resize(ulint hash_size)
{
 /* Step-1: Lock all search latches in exclusive mode. */
 btr_search_x_lock_all();
...
 /* Step-2: Recreate hash tables with new size. */
 for (ulint i = 0; i < btr_ahi_parts; ++i) {

  mem_heap_free(btr_search_sys->hash_tables[i]->heap);
  hash_table_free(btr_search_sys->hash_tables[i]);

  btr_search_sys->hash_tables[i] =
...
}

Similar reasoning applies in the next srv_export_innodb_status part dealing with dict sys hash tables.

[Laurynas Biveinis (laurynas-biveinis)]

https://github.com/percona/percona-server/pull/476

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-978