crash info written to relay.index or bin.index instead of error.log

[monty solomon]

Percona Server wrote the crash info to the relay.index file instead of to the error.log file and the slave was unable to start due to error 1872.

[ERROR] Slave SQL for channel '': Slave failed to initialize relay log info structure from the repository, Error_code: 1872

[monty solomon (monty+launchpad)]

This happened for the crash reported in bug 1618257 as well as other crashes.

# cat relay.index
/opt/mysql/dblogs1/monty-p0/relay/relay.002566
/opt/mysql/dblogs1/monty-p0/relay/relay.002567
/opt/mysql/dblogs1/monty-p0/relay/relay.002568
/opt/mysql/dblogs1/monty-p0/relay/relay.002569
/opt/mysql/dblogs1/monty-p0/relay/relay.002570
22:41:43 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=27
max_threads=701
thread_count=21
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 971371 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
/opt/mysql/dblogs1/monty-p0/relay/relay.002571

[monty solomon (monty+launchpad)]

mysql Ver 14.14 Distrib 5.7.13-6, for Linux (x86_64) using 6.0
Server version: 5.7.13-6-log Percona Server (GPL), Release 6, Revision e3d58bb
CentOS release 6.7 (Final)
Linux shaky-cat 3.18.36.el6.x86_64 #1 SMP Tue Aug 9 18:09:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[monty solomon (monty+launchpad)]

Sometimes it writes to the bin.index file

/opt/mysql/dblogs1/monty-q0/bin/bin.012648
22:10:44 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=9
max_threads=501
thread_count=7
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 712565 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[monty solomon (monty+launchpad)]

mysql Ver 14.14 Distrib 5.7.12-5, for Linux (x86_64) using 6.0
Server version: 5.7.12-5-log Percona Server (GPL), Release 5, Revision a2f663a
CentOS release 6.7 (Final)
Linux noisy-dawn 3.18.30-88.el6.x86_64 #1 SMP Thu Apr 7 23:47:59 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[monty solomon (monty+launchpad)]

mysqld: Error reading file '/opt/mysql/dblogs1/monty-q0/bin/' (Errcode: 21 - Is a directory)
2016-08-30T22:10:47.081179Z 0 [ERROR] I/O error reading the header from the binary log, errno=21, io cache code=-1
2016-08-30T22:10:47.081204Z 0 [ERROR] I/O error reading the header from the binary log
2016-08-30T22:10:47.081210Z 0 [ERROR] Can't init tc log
2016-08-30T22:10:47.081214Z 0 [ERROR] Aborting

[monty solomon (monty+launchpad)]

mysqld crashed on another server today and some output was written to bin.index and some was written to the error log. mysqld was unable to restart due to the output in bin.index.

In bin.index

09:46:00 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=90
max_threads=501
thread_count=69
connection_count=67
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 712569 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f7983931b30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f7a3e4e9d40 thread_stack 0x40000

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f7983e3ad30): is an invalid pointer
Connection ID (thread ID): 534038
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[monty solomon (monty+launchpad)]

In error.log

/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xec6b8c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x79dba1]
/lib64/libpthread.so.0[0x3e9360f7e0]
/usr/lib64/mysql/plugin/audit_log.so(+0x4de0)[0x7f7a64079de0]
/usr/lib64/mysql/plugin/audit_log.so(+0x67b5)[0x7f7a6407b7b5]
/usr/lib64/mysql/plugin/audit_log.so(+0x695f)[0x7f7a6407b95f]
/usr/sbin/mysqld[0x79dd93]
/usr/sbin/mysqld(_Z18mysql_audit_notifyP3THD30mysql_event_general_subclass_tiPKcm+0x1f4)[0x79e314]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x3ae)[0xcac2ee]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1b7)[0xcae207]
/usr/sbin/mysqld(handle_connection+0x2a0)[0xd6fd00]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xede504]
/lib64/libpthread.so.0[0x3e93607aa1]
/lib64/libc.so.6(clone+0x6d)[0x3e932e893d]
2016-09-07T09:46:01.917002Z mysqld_safe Number of processes running now: 0
2016-09-07T09:46:01.918718Z mysqld_safe mysqld restarted

[monty solomon (monty+launchpad)]

mysql Ver 14.14 Distrib 5.7.14-7, for Linux (x86_64) using 6.0
Server version: 5.7.14-7-log Percona Server (GPL), Release 7, Revision 083e298
CentOS release 6.7 (Final)

[monty solomon (monty+launchpad)]

This behavior seems to happen each time it crashes. Here is another example.

Written to the bin.index file

02:29:34 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=580
max_threads=701
thread_count=328
connection_count=328
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 971371 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[monty solomon (monty+launchpad)]

The bug status is still Undecided. Has this bug been triaged?

[monty solomon (monty+launchpad)]

The bug status is still New/Undecided. Has this bug been triaged?

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3540