5.5.15-rel21.0.158 crashes during replication, can't recover

[Eamon Daly]

Replicating from a 5.5.15-rel21.0.158 server to a 5.5.15-rel21.0.158 server crashed with the following error and attempt at recovery:

---
111012 14:12:27 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=1
max_threads=100
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1057963 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x17ae01d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x41b1d0f8 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7cdb99]
/usr/sbin/mysqld(handle_segfault+0x379)[0x500b59]
/lib64/libpthread.so.0[0x3a3260eb10]
/usr/sbin/mysqld(_ZNK9table_def15compatible_withEP3THDP14Relay_log_infoP5TABLEPS5_+0x31d)[0x7b143d]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xf9c)[0x74a0fc]
/usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14Relay_log_info+0x152)[0x516be2]
/usr/sbin/mysqld[0x5192f2]
/usr/sbin/mysqld(handle_slave_sql+0xc19)[0x5208f9]
/lib64/libpthread.so.0[0x3a3260673d]
/lib64/libc.so.6(clone+0x6d)[0x3a312d44bd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
---

InnoDB then went through its recovery process, then MySQL restarted and reported several MyISAM tables crashed. mysqlcheck cleared them as OK, but when replication was restarted it entered a crash loop. The binlog entry was:

---
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#111009 22:21:54 server id 3 end_log_pos 107 Start: binlog v 4, server v 5.5.15-55-log created 111009 22:21:54
BINLOG '
0mSSTg8DAAAAZwAAAGsAAAAAAAQANS41LjE1LTU1LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAVAAEGggAAAAICAgCAA==
'/*!*/;
# at 1074397168
#111012 13:41:24 server id 3 end_log_pos 1074397253 Query thread_id=680 exec_time=0 error_code=0
SET TIMESTAMP=1318444884/*!*/;
SET @@session.pseudo_thread_id=680/*!*/;
SET @@session.foreign_key_checks=0, @@session.sql_auto_is_null=0, @@session.unique_checks=0, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=524288/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=8/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
COMMIT
/*!*/;
# at 1074397253
#111012 13:41:24 server id 3 end_log_pos 1074397296 Rotate to mysql-bin.000074 pos: 4
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
---

The error log is attached.

[yinfeng (yinfeng-zwx)]

is this a duplicate bug of https://bugs.launchpad.net/percona-server/+bug/1019125 ?
and the fix for bug #47103(http://bugs.mysql.com/bug.php?id=47103) was not backported into 5.5 series.

[Alexey Kopytov (akopytov)]

Yes, marking as a duplicate of bug #1019125.

[Mario Splivalo (mariosplivalo)]

Although this seems to be fixed in recent 5.5 (5.5.31 as of this writing), and 5.6 (5.6.11), the same issue happens when using cascading replication setup - second slave in chain crashes with the same error as described above.
Filled separate bug report for that issue: https://bugs.launchpad.net/percona-server/+bug/1191263