mysqld crash

[Philip]

mysqld Ver 5.5.31-30.3-log for Linux on x86_64 (Percona Server (GPL), Release rel30.3, Revision 520)

uname -a
Linux db02.domain.com 3.4.37-40.44.amzn1.x86_64 #1 SMP Thu Mar 21 01:17:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Here is the detail from error.log

02:05:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=524288000
read_buffer_size=2097152
max_used_connections=4
max_threads=353
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1962208 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x13f887a0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f905e992da8 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7b4965]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x68f704]
/lib64/libpthread.so.0(+0xf8e0)[0x7f905e6c08e0]
/usr/sbin/mysqld[0x7e0774]
/usr/sbin/mysqld[0x5d0854]
/usr/sbin/mysqld[0x5e1a4b]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x25e)[0x5cff6e]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4dd)[0x5caa4d]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5cc95c]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5cd40d]
/usr/sbin/mysqld[0x5895d2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1553)[0x58e2d3]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x33b)[0x591adb]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x163d)[0x5931bd]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x13f)[0x62e4cf]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x62e5b1]
/lib64/libpthread.so.0(+0x7c6b)[0x7f905e6b8c6b]
/lib64/libc.so.6(clone+0x6d)[0x7f905d3175ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f8d480009a0): is an invalid pointer
Connection ID (thread ID): 67752
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
130607 02:05:04 mysqld_safe Number of processes running now: 0
130607 02:05:04 mysqld_safe mysqld restarted
130607 2:05:04 [Note] Plugin 'FEDERATED' is disabled.
130607 2:05:04 [Warning] option 'innodb-autoextend-increment': unsigned value 262144 adjusted to 1000
130607 2:05:04 InnoDB: The InnoDB memory heap is disabled
130607 2:05:04 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130607 2:05:04 InnoDB: Compressed tables use zlib 1.2.3
130607 2:05:04 InnoDB: Using Linux native AIO
130607 2:05:04 InnoDB: Initializing buffer pool, size = 10.0G

[Valerii Kravchuk (valerii-kravchuk)]

Are you able to identify crashing statement? I wonder if it accesses any tables in the INFORMATION_SCHEMA.

[Philip (philip-1)]

Valerii,

This server is a slave and does not have any direct queries (or should
not). What would be the best means to track down what was running just
prior the crash?

Thanks

On Fri, Jun 7, 2013 at 3:38 AM, Valerii Kravchuk <
<email address hidden>> wrote:

> Are you able to identify crashing statement? I wonder if it accesses any
> tables in the INFORMATION_SCHEMA.
>
> ** Changed in: percona-server
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1188447
>
> Title:
> mysqld crash
>
> Status in Percona Server with XtraDB:
> Incomplete
>
> Bug description:
> mysqld Ver 5.5.31-30.3-log for Linux on x86_64 (Percona Server (GPL),
> Release rel30.3, Revision 520)
>
> uname -a
> Linux db02.domain.com 3.4.37-40.44.amzn1.x86_64 #1 SMP Thu Mar 21
> 01:17:08 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> Here is the detail from error.log
>
> 02:05:02 UTC - mysqld got signal 11 ;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose the problem, but since we have already crashed,
> something is definitely wrong and this may fail.
> Please help us make Percona Server better by reporting any
> bugs at http://bugs.percona.com/
>
> key_buffer_size=524288000
> read_buffer_size=2097152
> max_used_connections=4
> max_threads=353
> thread_count=1
> connection_count=1
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
> 1962208 K bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> Thread pointer: 0x13f887a0
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> stack_bottom = 7f905e992da8 thread_stack 0x40000
> /usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7b4965]
> /usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x68f704]
> /lib64/libpthread.so.0(+0xf8e0)[0x7f905e6c08e0]
> /usr/sbin/mysqld[0x7e0774]
> /usr/sbin/mysqld[0x5d0854]
> /usr/sbin/mysqld[0x5e1a4b]
>
> /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x25e)[0x5cff6e]
> /usr/sbin/mysqld(_ZN4JOIN4execEv+0x4dd)[0x5caa4d]
>
> /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5cc95c]
>
> /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5cd40d]
> /usr/sbin/mysqld[0x5895d2]
> /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1553)[0x58e2d3]
> /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x33b)[0x591adb]
>
> /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x163d)[0x5931bd]
> /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x13f)[0x62e4cf]
> /usr/sbin/mysqld(...

[Valerii Kravchuk (valerii-kravchuk)]

Based on stack trace:

...
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x25e)[0x5cff6e]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4dd)[0x5caa4d]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5cc95c]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5cd40d]
/usr/sbin/mysqld[0x5895d2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1553)[0x58e2d3]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x33b)[0x591adb]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x163d)[0x5931bd]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x13f)[0x62e4cf]
...

it was SELECT that crashed. So it was not executed by replication SQL thread, rather somebody connected to this server and executed SELECT. If you do NOT have general query log enabled you hardly can find what exact query it was. Probably you should try to check other logs in the system (syslog, /var/log/messages, any application level logs if you have applications connected to this slave) for any suspicious activity/connections at the moment of crash.

Please, send your my.cnf file content also. I wonder if core file could be created during the crash.

[Launchpad Janitor (janitor)]

[Expired for Percona Server because there has been no activity for 60 days.]

[Przemek (pmalkowski)]

Another similar crash was observed on a slave that was stuck with status variable Slave_open_temp_tables = 1 all the time. After following statement was run:
select * from information_schema.global_temporary_tables;
the server crashed.
Version: Percona Server 5.5.36-34.2-34.2

Prior to crash, the temp table was confirm to exist on disk:
-rw-rw---- 1 mysql mysql 8568 Sep 23 15:07 #sql1a67_3ca8e30a_7a.frm
-rw-rw---- 1 mysql mysql 98304 Sep 23 15:08 #sql1a67_3ca8e30a_7a.ibd

Thread pointer: 0x1335fb400
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fdca36a1d78 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7c8425]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x69d824]
/lib64/libpthread.so.0[0x32fe80f500]
/usr/sbin/mysqld[0x82c166]
/usr/sbin/mysqld[0x5da0c4]
/usr/sbin/mysqld[0x5eacac]
/usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x25e)[0x5d97de]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x4dd)[0x5d424d]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x12c)[0x5d612c]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5d6bdd]
/usr/sbin/mysqld[0x590b5a]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1f1a)[0x59632a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x33b)[0x59937b]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1eb2)[0x59bb02]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x13f)[0x63a5ff]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x63a6e1]
/lib64/libpthread.so.0[0x32fe807851]
/lib64/libc.so.6(clone+0x6d)[0x32fe4e894d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (d2da8010): select * from information_schema.global_temporary_tables
Connection ID (thread ID): 1385568033
Status: NOT_KILLED

(...)
141024 12:47:00 [Note] Starting crash recovery...
141024 12:47:00 [Note] Crash recovery finished.
141024 12:47:00 InnoDB: Error: table `tmp`.`#sql1a67_3ca8e30a_7a` does not
exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.

[Muhammad Irfan (muhammad-irfan)]

@Philip,

This seems quite similar to https://bugs.launchpad.net/percona-server/+bug/1294190
This is fixed in Percona Server 5.6.22-71.0 Can you please upgrade and check If that resolves the problem. Thanks.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2970