mysqld crash double free or corruption

[monty solomon]

Linux hymes 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

CentOS release 6.5 (Final)

mysql Ver 14.14 Distrib 5.6.19-67.0, for Linux (x86_64) using EditLine wrapper

Server version: 5.6.19-67.0-log Percona Server (GPL), Release 67.0, Revision 618

mysqld crashed three (3) times.

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (fasttop): 0x00007f60c9f07130 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7f648a2cc166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7f6089702c74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7f60896f84b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7f608991ea8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7f6089921428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7f60899216c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7f60899226f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7f60899229fc]
/lib64/libpam.so.0(+0x2cee)[0x7f648801bcee]
/lib64/libpam.so.0(pam_authenticate+0x40)[0x7f648801b600]
/usr/lib64/mysql/plugin/auth_pam.so(authenticate_user_with_pam_server+0xda)[0x7f644ec6e55a]
/usr/sbin/mysqld[0x660ddd]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0xf11)[0x675f61]
/usr/sbin/mysqld[0x6a2393]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x6a3de2]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6a4534]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x6a4856]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6a4970]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0xb0dc13]
/lib64/libpthread.so.0(+0x79d1)[0x7f648bc1f9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f648a33eb6d]

08:03:54 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=326
max_threads=502
thread_count=112
connection_count=112
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 715477 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x15e26c60
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f608ba40d40 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8bb80c]
/usr/sbin/mysqld(handle_fatal_signal+0x479)[0x6504b9]
/lib64/libpthread.so.0(+0xf710)[0x7f648bc27710]
/lib64/libc.so.6(gsignal+0x35)[0x7f648a288925]
/lib64/libc.so.6(abort+0x175)[0x7f648a28a105]
/lib64/libc.so.6(+0x70837)[0x7f648a2c6837]
/lib64/libc.so.6(+0x76166)[0x7f648a2cc166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7f6089702c74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7f60896f84b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7f608991ea8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7f6089921428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7f60899216c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7f60899226f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7f60899229fc]
/lib64/libpam.so.0(+0x2cee)[0x7f648801bcee]
/lib64/libpam.so.0(pam_authenticate+0x40)[0x7f648801b600]
/usr/lib64/mysql/plugin/auth_pam.so(authenticate_user_with_pam_server+0xda)[0x7f644ec6e55a]
/usr/sbin/mysqld[0x660ddd]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0xf11)[0x675f61]
/usr/sbin/mysqld[0x6a2393]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x6a3de2]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6a4534]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x6a4856]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6a4970]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0xb0dc13]
/lib64/libpthread.so.0(+0x79d1)[0x7f648bc1f9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f648a33eb6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 83936106
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
141023 08:03:56 mysqld_safe Number of processes running now: 0
141023 08:03:56 mysqld_safe mysqld restarted

[monty solomon (monty+launchpad)]

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (fasttop): 0x00007f08d8009230 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7f0cb3bfc166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7f0985e8ec74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7f0985e844b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7f09860aaa8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7f09860ad428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7f09860ad6c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7f09860ae6f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7f09860ae9fc]
/lib64/libpam.so.0(+0x2cee)[0x7f0cb12eacee]
/lib64/libpam.so.0(pam_authenticate+0x40)[0x7f0cb12ea600]
/usr/lib64/mysql/plugin/auth_pam.so(authenticate_user_with_pam_server+0xda)[0x7f0c772cf55a]
/usr/sbin/mysqld[0x660ddd]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0xf11)[0x675f61]
/usr/sbin/mysqld[0x6a2393]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x6a3de2]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6a4534]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x6a4856]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6a4970]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0xb0dc13]
/lib64/libpthread.so.0(+0x79d1)[0x7f0cb554f9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f0cb3c6eb6d]

08:40:41 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=254
max_threads=502
thread_count=185
connection_count=185
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 715477 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x15077b40
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f092b2d5d40 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8bb80c]
/usr/sbin/mysqld(handle_fatal_signal+0x479)[0x6504b9]
/lib64/libpthread.so.0(+0xf710)[0x7f0cb5557710]
/lib64/libc.so.6(gsignal+0x35)[0x7f0cb3bb8925]
/lib64/libc.so.6(abort+0x175)[0x7f0cb3bba105]
/lib64/libc.so.6(+0x70837)[0x7f0cb3bf6837]
/lib64/libc.so.6(+0x76166)[0x7f0cb3bfc166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7f0985e8ec74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7f0985e844b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7f09860aaa8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7f09860ad428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7f09860ad6c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7f09860ae6f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7f09860ae9fc]
/lib64/libpam.so.0(+0x2cee)[0x7f0cb12eacee]
/lib64/libpa...

[monty solomon (monty+launchpad)]

*** glibc detected *** /usr/sbin/mysqld: double free or corruption (fasttop): 0x00007fe33047f040 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x76166)[0x7fe71c04c166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7fe3edfcfc74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7fe3edfc54b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7fe3ee1eba8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7fe3ee1ee428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7fe3ee1ee6c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7fe3ee1ef6f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7fe3ee1ef9fc]
/lib64/libpam.so.0(+0x2cee)[0x7fe71973acee]
/lib64/libpam.so.0(pam_authenticate+0x40)[0x7fe71973a600]
/usr/lib64/mysql/plugin/auth_pam.so(authenticate_user_with_pam_server+0xda)[0x7fe6df2cf55a]
/usr/sbin/mysqld[0x660ddd]
/usr/sbin/mysqld(_Z16acl_authenticateP3THDj+0xf11)[0x675f61]
/usr/sbin/mysqld[0x6a2393]
/usr/sbin/mysqld(_Z16login_connectionP3THD+0x42)[0x6a3de2]
/usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x6a4534]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x6a4856]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6a4970]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0xb0dc13]
/lib64/libpthread.so.0(+0x79d1)[0x7fe71d99f9d1]
/lib64/libc.so.6(clone+0x6d)[0x7fe71c0beb6d]

12:09:07 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=291
max_threads=502
thread_count=132
connection_count=132
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 715477 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x160c1cb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe339851d40 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8bb80c]
/usr/sbin/mysqld(handle_fatal_signal+0x479)[0x6504b9]
/lib64/libpthread.so.0(+0xf710)[0x7fe71d9a7710]
/lib64/libc.so.6(gsignal+0x35)[0x7fe71c008925]
/lib64/libc.so.6(abort+0x175)[0x7fe71c00a105]
/lib64/libc.so.6(+0x70837)[0x7fe71c046837]
/lib64/libc.so.6(+0x76166)[0x7fe71c04c166]
/lib64/libldap-2.4.so.2(ldap_pvt_tls_set_option+0x244)[0x7fe3edfcfc74]
/lib64/libldap-2.4.so.2(ldap_set_option+0xd6)[0x7fe3edfc54b6]
/lib64/security/pam_ldap.so(+0x2a8f)[0x7fe3ee1eba8f]
/lib64/security/pam_ldap.so(+0x5428)[0x7fe3ee1ee428]
/lib64/security/pam_ldap.so(+0x56c1)[0x7fe3ee1ee6c1]
/lib64/security/pam_ldap.so(+0x66f3)[0x7fe3ee1ef6f3]
/lib64/security/pam_ldap.so(pam_sm_authenticate+0x2fc)[0x7fe3ee1ef9fc]
/lib64/libpam.so.0(+0x2cee)[0x7fe71973acee]
/lib64/libpa...

[Valerii Kravchuk (valerii-kravchuk)]

Please, check if the same problem still happens with a newer version, 5.6.21. Also it would be nice to get more details on your LDAP authentication setup for MySQL via plugin.

[monty solomon (monty+launchpad)]

Upgraded the instances to 5.6.21.

[monty solomon (monty+launchpad)]

LDAP authentication setup

# cat /etc/pam.d/mysql
#%PAM-1.0
auth required pam_warn.so
auth required pam_ldap.so
account required pam_ldap.so

[Muhammad Irfan (muhammad-irfan)]

We need further information on this. Can you please identify on which conditions mysql server gets crash i.e. what triggers mysql to crash. Also, please provide my.cnf file output and "SHOW GLOBAL STATUS;" output will helpful too.
Further, I noticed /etc/pam.d/mysql which is for Oracle PAM plugin while /etc/pam.d/mysqld for Percona PAM plugin. Are you using Oracle MySQL PAM plugin or Percona PAM ?

[monty solomon (monty+launchpad)]

We are running Percona Server and using the Percona PAM plugin.

I specified mysql as the service name in the authentication_string of the GRANT statement and used mysql as the name of the file in the pam.d directory. The GRANT statement contained IDENTIFIED WITH auth_pam AS 'mysql';

I used mysql for the service name because the name of the MySQL service on RHEL/CentOS when running Percona is mysql. The name of the MySQL service on RHEL/CentOS when running Oracle MySQL is mysqld.

CentOS release 6.5 (Final)

$ chkconfig --list | grep sql
mysql 0:off 1:off 2:on 3:on 4:on 5:on 6:off

mysql Ver 14.14 Distrib 5.6.21-70.0, for Linux (x86_64) using EditLine wrapper
Server version: 5.6.21-70.0-log Percona Server (GPL), Release 70.0, Revision 688

mysql> pager grep pam
PAGER set to 'grep pam'
mysql> show plugins;
| auth_pam | ACTIVE | AUTHENTICATION | auth_pam.so | GPL |
| auth_pam_compat | ACTIVE | AUTHENTICATION | auth_pam_compat.so | GPL |

The Percona PAM Plugin documentation does not provide any guidance or explanation for configuring the plugin to be used with LDAP.

[monty solomon (monty+launchpad)]

I am not sure what triggered the crash. It appeared to be related to using PAM after upgrading to Percona 5.6.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3216