Crash in update_global_table_stats

[Sveta Smirnova]

Percona server 5.5.28-rel29.1-log crashes each time on startup with following stack trace:

130513 11:44:20 mysqld_safe Starting mysqld daemon with databases from /home/mysqladm/mysql/data
130513 11:44:20 [Warning] Changed limits: max_open_files: 1024 max_connections: 100 table_cache: 457
130513 11:44:20 [Note] Plugin 'FEDERATED' is disabled.
130513 11:44:20 InnoDB: The InnoDB memory heap is disabled
130513 11:44:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130513 11:44:20 InnoDB: Compressed tables use zlib 1.2.3
130513 11:44:20 InnoDB: Using Linux native AIO
130513 11:44:20 InnoDB: Initializing buffer pool, size = 2.0G
130513 11:44:21 InnoDB: Completed initialization of buffer pool
130513 11:44:21 InnoDB: highest supported file format is Barracuda.
[mysqladm@dbsrs0058:/home/mysqladm/mysql/log] cat mysqld.log
150525 01:36:55 mysqld_safe Starting mysqld daemon with databases from /home/mysqladm/mysql/data
150525 1:36:55 [Warning] Changed limits: max_open_files: 1024 max_connections: 100 table_cache: 457
150525 1:36:55 [Note] Plugin 'FEDERATED' is disabled.
06:36:55 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=125829120
read_buffer_size=2097152
max_used_connections=0
max_threads=100
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 533705 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fff6e1811e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fff6e1842f0 thread_stack 0x40000
/home/mysqladm/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x7d9795]
/home/mysqladm/mysql/bin/mysqld(handle_fatal_signal+0x3e1)[0x695d51]
/lib64/libpthread.so.0[0x3985e0eb10]
/home/mysqladm/mysql/bin/mysqld(my_hash_insert+0x25c)[0x7c23dc]
/home/mysqladm/mysql/bin/mysqld(_ZN7handler25update_global_table_statsEv+0x2ba)[0x69cdaa]
/home/mysqladm/mysql/bin/mysqld(_Z18close_thread_tableP3THDPP5TABLE+0xa1)[0x5460c1]
/home/mysqladm/mysql/bin/mysqld(_Z19close_thread_tablesP3THD+0x16f)[0x5469bf]
/home/mysqladm/mysql/bin/mysqld(_Z18close_mysql_tablesP3THD+0x18)[0x546a48]
/home/mysqladm/mysql/bin/mysqld(_Z11plugin_initPiPPci+0x116d)[0x58d41d]
/home/mysqladm/mysql/bin/mysqld[0x506e18]
/home/mysqladm/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x8d8)[0x509f58]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x398561d994]
/home/mysqladm/mysql/bin/mysqld[0x500d29]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
150525 01:36:55 mysqld_safe mysqld from pid file /home/mysqladm/tmp/mysqld.pid ended

Server cannot start, crash started to happen after normal shutdown and following restart of virtual machine where mysqld was running

Workaround: disable userstat

I cannot repeat this yet.

[Roel Van de Paar (roel11)]

There may be a connection with bug 1417382.

Whilst now the testcase from bug 1417382 produces different results, originally it must have resulted in a (error log) _ZN7handler25update_global_table_statsEv stack trace, as this is how it was classified in our bug string search list:

ZN7handler25update_global_table_statsEv ## http://bugs.mysql.com/bug.php?id=75741 (LP PS#1417382)

[Roel Van de Paar (roel11)]

The testcase now (on more recent versions) produces different stacks pointing to other logged bugs. Seems like a myriad of interconnected bugs.

[Roel Van de Paar (roel11)]

Actually, it may be that due to the different startup options different bugs are produced.

[Sveta Smirnova (svetasmirnova)]

Test case for bug #1417382 is not repeatable with version 5.5.28

[Valerii Kravchuk (valerii-kravchuk)]

Can we get my.cnf settings here and number of tables per storage engine used?

I wonder if this may be related to some specific storage engine used for some table, but not enabled upon startup.

[Sveta Smirnova (svetasmirnova)]

my.cnf is file #1 in the linked issue. According to additional information customer did not provide information about storage engines used.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3287