mysql crashed during shutdown

[monty solomon]

Executed the command service mysql restart

In the error.log file

2016-09-22T04:00:28.738785Z 4 [Note] Slave I/O thread exiting for channel '', read up to log 'bin.001893', position 731896422
2016-09-22T04:00:28.765019Z 0 [Note] Giving 6 client threads a chance to die gracefully
2016-09-22T04:00:28.765043Z 0 [Note] Shutting down slave threads
2016-09-22T04:00:30.770422Z 0 [Note] Forcefully disconnecting 2 remaining clients
2016-09-22T04:00:30.770484Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 2827645 user: 'SUSR_Monitor'

2016-09-22T04:00:30.770531Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 2826709 user: 'SUSR_Monitor'

terminate called without an active exception
2016-09-22T04:00:30.771406Z 0 [Note] Event Scheduler: Purging the queue. 0 events
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xec501c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79d141]
/lib64/libpthread.so.0(+0xf7e0)[0x7f9562ae07e0]
/lib64/libc.so.6(gsignal+0x35)[0x7f9560d1a625]
/lib64/libc.so.6(abort+0x175)[0x7f9560d1be05]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7f95615d4a7d]
/usr/lib64/libstdc++.so.6(+0xbcbd6)[0x7f95615d2bd6]
/usr/lib64/libstdc++.so.6(+0xbcc03)[0x7f95615d2c03]
/usr/lib64/libstdc++.so.6(+0xbd55f)[0x7f95615d355f]
/usr/sbin/mysqld(_ZN19Audit_error_handler16handle_conditionEP3THDjPKcPN13Sql_condition19enum_severity_levelES3_+0x26)[0x79f6c6]
/usr/sbin/mysqld(_ZN3THD16handle_conditionEjPKcPN13Sql_condition19enum_severity_levelES1_+0x52)[0xc5f632]
/usr/sbin/mysqld(_ZN3THD15raise_conditionEjPKcN13Sql_condition19enum_severity_levelES1_+0x60)[0xc5f720]
/usr/sbin/mysqld(my_message_sql+0x5f)[0x78e13f]
/usr/sbin/mysqld(my_error+0xe0)[0xebf480]
/usr/sbin/mysqld[0xbfe37b]
/usr/sbin/mysqld[0xbfe604]
/usr/sbin/mysqld(my_net_read+0x2fc)[0xbff43c]
/usr/sbin/mysqld(_ZN16Protocol_classic11read_packetEv+0x1c)[0xc0aebc]
/usr/sbin/mysqld(_ZN16Protocol_classic11get_commandEP8COM_DATAP19enum_server_command+0x1b)[0xc0ad4b]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xce)[0xcac8de]
/usr/sbin/mysqld(handle_connection+0x2a8)[0xd6de28]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xedc844]
/lib64/libpthread.so.0(+0x7aa1)[0x7f9562ad8aa1]
/lib64/libc.so.6(clone+0x6d)[0x7f9560dd093d]
2016-09-22T04:00:32.793156Z mysqld_safe Number of processes running now: 0
2016-09-22T04:00:32.794417Z mysqld_safe mysqld restarted

In the bin.index file

pure virtual method called
04:00:30 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=49
max_threads=701
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 971371 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f8e500e31c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f8ee0875d40 thread_stack 0x40000

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 2827645
Status: KILL_CONNECTION

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[monty solomon (monty+launchpad)]

mysql Ver 14.14 Distrib 5.7.13-6, for Linux (x86_64) using 6.0

Server version: 5.7.13-6-log Percona Server (GPL), Release 6, Revision e3d58bb

CentOS release 6.7 (Final)

Linux huge-wasp 3.18.36-246.el6.x86_64 #1 SMP Tue Aug 9 18:09:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[monty solomon (monty+launchpad)]

A crash tonight on another server during mysql shutdown.

terminate called without an active exception
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xec501c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79d141]
/lib64/libpthread.so.0(+0xf7e0)[0x7ff3569077e0]
/lib64/libc.so.6(gsignal+0x35)[0x7ff354b41625]
/lib64/libc.so.6(abort+0x175)[0x7ff354b42e05]
/usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d)[0x7ff3553fba7d]
/usr/lib64/libstdc++.so.6(+0xbcbd6)[0x7ff3553f9bd6]
/usr/lib64/libstdc++.so.6(+0xbcc03)[0x7ff3553f9c03]
/usr/lib64/libstdc++.so.6(+0xbd55f)[0x7ff3553fa55f]
/usr/sbin/mysqld(_ZN19Audit_error_handler16handle_conditionEP3THDjPKcPN13Sql_condition19enum_severity_levelES3_+0x26)[0x79f6c6]
/usr/sbin/mysqld(_ZN3THD16handle_conditionEjPKcPN13Sql_condition19enum_severity_levelES1_+0x52)[0xc5f632]
/usr/sbin/mysqld(_ZN3THD15raise_conditionEjPKcN13Sql_condition19enum_severity_levelES1_+0x60)[0xc5f720]
/usr/sbin/mysqld(my_message_sql+0x5f)[0x78e13f]
/usr/sbin/mysqld(my_error+0xe0)[0xebf480]
/usr/sbin/mysqld[0xbfe37b]
/usr/sbin/mysqld[0xbfe604]
/usr/sbin/mysqld(my_net_read+0x2fc)[0xbff43c]
/usr/sbin/mysqld(_ZN16Protocol_classic11read_packetEv+0x1c)[0xc0aebc]
/usr/sbin/mysqld(_ZN16Protocol_classic11get_commandEP8COM_DATAP19enum_server_command+0x1b)[0xc0ad4b]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xce)[0xcac8de]
/usr/sbin/mysqld(handle_connection+0x2a8)[0xd6de28]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xedc844]
/lib64/libpthread.so.0(+0x7aa1)[0x7ff3568ffaa1]
/lib64/libc.so.6(clone+0x6d)[0x7ff354bf793d]
2016-09-23T00:03:51.899284Z mysqld_safe Number of processes running now: 0
2016-09-23T00:03:51.901035Z mysqld_safe mysqld restarted

pure virtual method called
00:03:49 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=244
max_threads=501
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 712565 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fea61fa6060
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7feb972cad40 thread_stack 0x40000

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 2106862
Status: KILL_CONNECTION

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the cra...

[monty solomon (monty+launchpad)]

Also running 5.7.13

mysql Ver 14.14 Distrib 5.7.13-6, for Linux (x86_64) using 6.0

Server version: 5.7.13-6-log Percona Server (GPL), Release 6, Revision e3d58bb

CentOS release 6.7 (Final)

Linux swift-sheep 3.18.36-246.el6.x86_64 #1 SMP Tue Aug 9 18:09:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[monty solomon (monty+launchpad)]

The bug status is still Undecided. Has this bug been triaged?

[Sergei Glushchenko (sergei.glushchenko)]

I can suggest that may be push_internal_handler/pop_internal_handler got unbalanced somehow. Is there steady procedure to reproduce this crash?

[monty solomon (monty+launchpad)]

It has been intermittent and we don't restart often.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3565