Crash mysqld segfault

[Rob]

Percona Server 5.5.28-rel29.1 crashed unexpectedly.

There was enough RAM available, disks were OK, load was not extraordinary (about 2500 q/s).

System:
2x Xeon E5620
32 GB RAM
4x SAS 300GB SCSI in RAID10 (Areca 1212)

We've upgraded to 5.5.42-37.1, so I'm not able to perform any test anymore, but it seemed like a good idea to report it anyway.

As to me this seems like a segfault during an incoming network connection, I have tagged it as a security issue.

12:15:02 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=262144
max_used_connections=501
max_threads=500
thread_count=210
connection_count=193
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25758704 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x26fa7d20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fd53f73aec0 thread_stack 0x40000
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x7d9795]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x3e1)[0x695d51]
/lib64/libpthread.so.0(+0x10660)[0x7fdb0d6ed660]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(_Z14ip_to_hostnameP16sockaddr_storagePKcPPcPj+0x14d)[0x69e67d]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(_Z16login_connectionP3THD+0x392)[0x621fd2]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x622134]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(_Z24do_handle_one_connectionP3THD+0xee)[0x6223ee]
/usr/local/Percona-Server-5.5.28-rel29.1-334.Linux.x86_64/bin/mysqld(handle_one_connection+0x54)[0x622504]
/lib64/libpthread.so.0(+0x83b4)[0x7fdb0d6e53b4]
/lib64/libc.so.6(clone+0x6d)[0x7fdb0c8d15ad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 192000064
Status: NOT_KILLED

[Valerii Kravchuk (valerii-kravchuk)]

Please, send your my.cnf file content and (if possible) the output of pt-summary script from this host.

Is it possible that (reverse) DNS lookup did not work properly (or fast enough) when this crash happened?

[Rob (rdewit-6)]

I cannot rule out a failing reverse DNS lookup, but all connections should be from within our local net where all hosts are known by /etc/hosts.

Here's my my.cnf and output of pt-summary of our current running server.

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 1G
max_allowed_packet = 32M
user=sql
datadir=/usr/local/mysql/data
tmpdir=/usr/local/mysql/var

read_buffer_size = 256K
sort_buffer_size = 48M
read_rnd_buffer_size = 256K
join_buffer_size = 16M

wait_timeout = 3

tmp_table_size = 256M
max_heap_table_size = 256M

thread_cache_size = 8
query_cache_size = 128M
query_cache_limit = 16M
max_connections = 500
max_join_size = 4G

memlock

ft_min_word_len = 2

myisam_sort_buffer_size = 512M

open_files_limit=65535
table_open_cache=4096

log_error = datahost.err
slow_query_log = off
pid_file = /var/run/mysql.pid
server_id = 304
expire_logs_days= 1
innodb_locks_unsafe_for_binlog = 1

innodb_data_home_dir = /usr/local/mysql/innodb/data/
innodb_data_file_path = ibdata1:100M:autoextend
innodb_log_group_home_dir = /usr/local/mysql/innodb/var/
innodb_mirrored_log_groups = 1
innodb_log_files_in_group = 3
innodb_log_file_size = 1G
innodb_log_buffer_size = 32M
innodb_buffer_pool_size = 20G
innodb_additional_mem_pool_size = 16M
innodb_lock_wait_timeout = 300

innodb_file_per_table

innodb_io_capacity = 750

innodb_flush_log_at_trx_commit = 1

tmpdir = /usr/local/mysql/var

loose_handlersocket_port = 9998
loose_handlersocket_port_wr = 9999
loose_handlersocket_threads = 16
loose_handlersocket_threads_wr = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer_size = 256M
sort_buffer = 256M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer_size = 512M
sort_buffer = 512M
read_buffer = 16M
write_buffer = 16M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
pid-file = /var/run/mysql/mysqld.pid
open_files_limit=32768

# Percona Toolkit System Summary Report ######################
        Date | 2015-03-17 15:44:28 UTC (local TZ: CET +0100)
    Hostname | datahost
      Uptime | 176 days, 5:23, 3 users, load average: 11.02, 11.60, 11.85
      System | Supermicro; X8DTU; v1234567890 (Main Server Chassis)
 Service Tag | 1234567890
    Platform | Linux
     Release | NAME=Slackware
      Kernel | 3.2.45
Architecture | CPU = 64-bit, OS = 64-bit
   Threading | NPTL 2.20
    Compiler | GNU CC version 4.8.3.
     SELinux | No SELinux detected
 Virtualized | No virtualization detected
# Processor ##################################################
  Processors | physical = 2, cores = 8, virtual = 16, hyperthreading = yes
      Speeds | 16x2399.824
      Models | 16xIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
      Caches | 16x12288 KB
# Memory #####################################################
       Total | 31.4G
        Free | 1.3G
        Used | physical = 30.1G, swap allocated = 7.6G, swap used = 247.1M, virtual = 30.3G
     Buffers | 4.6M
      Caches | 2.6G
       Dirty | 1032 kB
     UsedRSS | 26.6G
  Swappiness | 60
 DirtyPolicy | 20, 10
 DirtyStatus | 0, 0
  Locator Size Speed Form Factor ...

[Rob (rdewit-6)]

Just hit another one with the newest version 5.5.42-37.1:

13:01:01 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=262144
max_used_connections=501
max_threads=502
thread_count=159
connection_count=142
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 25857615 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x26168750
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff59562ae70 thread_stack 0x40000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0x7b94ce]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x491)[0x69ffd1]
/lib64/libpthread.so.0(+0x10660)[0x7ffb40b6b660]
/usr/local/mysql/bin/mysqld(_Z14ip_to_hostnameP16sockaddr_storagePKcPPcPj+0x15b)[0x6a8a9b]
/usr/local/mysql/bin/mysqld[0x646d9f]
/usr/local/mysql/bin/mysqld(_Z16login_connectionP3THD+0x50)[0x6487f0]
/usr/local/mysql/bin/mysqld(_Z22thd_prepare_connectionP3THD+0x24)[0x648ea4]
/usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x10f)[0x64919f]
/usr/local/mysql/bin/mysqld(handle_one_connection+0x51)[0x6492c1]
/lib64/libpthread.so.0(+0x83b4)[0x7ffb40b633b4]
/lib64/libc.so.6(clone+0x6d)[0x7ffb3f60f5ad]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 28560020
Status: NOT_KILLED

[Rob (rdewit-6)]

In reference to the old thread on http://bugs.mysql.com/bug.php?id=57945 that seems to describe the same problem we have., we also have a crontab that flushes hosts every minute. Could that be of any influence?

[Rob (rdewit-6)]

FYI: After modifying our privileges accordingly, we have restarted the server with skip_name_resolve nad no crashes have occurred ever since.

This seems to be a valid workaround to us.

[George Ormond Lorch III (gl-az)]

IS it possible that you have a host with a long hostname (> 0 characters) https://bugs.launchpad.net/percona-server/+bug/1431054

[George Ormond Lorch III (gl-az)]

I meant > 60 characters

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3274