Percona Server Version: 5.6.22-71.0 - mysqld got signal 11

[Ravi Ramamoorthy]

20:39:36 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=53
max_threads=4002
thread_count=52
connection_count=49
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3148778 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2aceebd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff7ebb6ce30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x90aa6c]
/usr/sbin/mysqld(handle_fatal_signal+0x352)[0x6643b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7ffe9a396cb0]
/lib/x86_64-linux-gnu/libc.so.6(+0x149e68)[0x7ffe99b15e68]
/usr/sbin/mysqld(_ZN6String4copyEPKcjPK15charset_info_st+0x7e)[0x739bfe]
/usr/sbin/mysqld[0x8c29d6]
/usr/sbin/mysqld(_Z15com_binlog_dumpP3THDPcj+0x1d9)[0x8c5a59]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x10ae)[0x6f8d2e]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x20d)[0x6bf27d]
/usr/sbin/mysqld(handle_one_connection+0x39)[0x6bf2f9]
/usr/sbin/mysqld(pfs_spawn_thread+0x140)[0xb681c0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7ffe9a38ee9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffe99ac02ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 29684031
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

------------
Ubuntu version: uname -a
Linux dbmaster01 3.8.0-44-generic #66~precise1-Ubuntu SMP Tue Jul 15 04:01:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Memory: MemTotal: 65976816 kB

[Nilnandan Joshi (nilnandan-joshi)]

Hi, Initially it seems, mysqld was crashed due to some bad events/entry in binlog file.

/usr/sbin/mysqld(handle_fatal_signal+0x352)[0x6643b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7ffe9a396cb0]
/lib/x86_64-linux-gnu/libc.so.6(+0x149e68)[0x7ffe99b15e68]
/usr/sbin/mysqld(_ZN6String4copyEPKcjPK15charset_info_st+0x7e)[0x739bfe]
/usr/sbin/mysqld[0x8c29d6]
/usr/sbin/mysqld(_Z15com_binlog_dumpP3THDPcj+0x1d9)[0x8c5a59]

Can you please provide some more information ? like

1. my.cnf
2. full error log
3. mysqlbinlog output from corresponding IO position where you got above error.

[Ravi Ramamoorthy (rramamoorthy)]

Hi,

I have attached below required files from the crashed db server dbmaster01
which is serving as master/slave dual role supporting 50 replications slave
servers
1. my.cnf
2. dbmaster01-error.log
3. mysqlbinlog output
file mysqlbinlog_bin_000067-position_81214636-output.txt
   (found in the error.log file after crash "InnoDB: Last MySQL binlog file
position 0 81214636, file name mysql-bin.000067"

​Thanks,
Ravi
​ Ramamoorthy
Sr. Database Admin​

On Tue, Feb 17, 2015 at 12:27 AM, Nilnandan Joshi <
<email address hidden>> wrote:

> Hi, Initially it seems, mysqld was crashed due to some bad events/entry
> in binlog file.
>
> /usr/sbin/mysqld(handle_fatal_signal+0x352)[0x6643b2]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7ffe9a396cb0]
> /lib/x86_64-linux-gnu/libc.so.6(+0x149e68)[0x7ffe99b15e68]
> /usr/sbin/mysqld(_ZN6String4copyEPKcjPK15charset_info_st+0x7e)[0x739bfe]
> /usr/sbin/mysqld[0x8c29d6]
> /usr/sbin/mysqld(_Z15com_binlog_dumpP3THDPcj+0x1d9)[0x8c5a59]
>
> Can you please provide some more information ? like
>
> 1. my.cnf
> 2. full error log
> 3. mysqlbinlog output from corresponding IO position where you got above
> error.
>
> ** Changed in: percona-server
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1420986
>
> Title:
> Percona Server Version: 5.6.22-71.0 - mysqld got signal 11
>
> Status in Percona Server with XtraDB:
> Incomplete
>
> Bug description:
> 20:39:36 UTC - mysqld got signal 11 ;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose the problem, but since we have already crashed,
> something is definitely wrong and this may fail.
> Please help us make Percona Server better by reporting any
> bugs at http://bugs.percona.com/
>
> key_buffer_size=16777216
> read_buffer_size=262144
> max_used_connections=53
> max_threads=4002
> thread_count=52
> connection_count=49
> It is possible that mysqld could use up to
> key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
> 3148778 K bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> Thread pointer: 0x2aceebd0
> Attempting backtrace. You can use the following information to find out
> where mysqld died. If you see no messages after this, something went
> terribly wrong...
> stack_bottom = 7ff7ebb6ce30 thread_stack 0x40000
> /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x90aa6c]
> /usr/sbin/mysqld(handle_fatal_signal+0x352)[0x6643b2]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7ffe9a396cb0]
> /lib/x86_64-linux-gnu/libc.so.6(+0x149e68)[0x7ffe99b15e68]
> /usr/sbin/mysqld(_ZN6String4copyEPKcjPK15charset_info_st+0x7e)[0x739bfe]
> /usr/sbin/mysqld[0x8c29d6]
> /usr/sbin/mysqld(_Z15com_binlog_dumpP3THDPcj+0x1d9)[0x8c5a59]
>
> /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDP...

[Launchpad Janitor (janitor)]

[Expired for Percona Server because there has been no activity for 60 days.]

[Nilnandan Joshi (nilnandan-joshi)]

Hi,

I don't see anything suspicious in binlog file but, I found below warning in error log even sometime just before the crash.

2015-02-12 02:23:35 6410 [Warning] Slave I/O: Notifying master by SET @master_binlog_checksum= @@global.binlog_checksum failed with error: Unknown system variable 'binlog_checksum', Error_code: 1193
2015-02-12 02:23:35 6410 [Warning] Slave I/O: Unknown system variable 'SERVER_UUID' on master. A probable cause is that the variable is not supported on the master (version: 5.5.34-log), even though it is on the slave (version: 5.6.22-71.0-log), Error_code: 1193

It seems, your master is 5.5 and slave 5.6. I'm not sure but this can be the reason for crash. Are you planning to upgrade master?

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.5 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.6 because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3262