Crash during InnoDB compression recovery

[monty solomon]

Started up a Percona 5.6 instance using an EBS snapshot of a Percona 5.5 instance that has compressed tables and got a crash due to assertion failure in file page0zip.cc line 4357

New
Percona-Server-shared-56-5.6.16-rel64
Linux 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 6.5 (Final)

Old
Percona-Server-shared-55-5.5.35-rel33.0.611
Linux 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
CentOS release 6.5 (Final)

[monty solomon (monty+launchpad)]

InnoDB: Doing recovery: scanned up to log sequence number 147368420225
2014-03-21 12:58:41 8061 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 2014-03-21 12:58:48 7fb4d4f56700 InnoDB: Assertion failure in thread 140414643693312 in file page0zip.cc line 4357
InnoDB: Failing assertion: slot_rec
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:58:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=0
max_threads=502
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1614987 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8ca6d5]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x645024]
/lib64/libpthread.so.0(+0xf710)[0x7fb7f5cca710]
/lib64/libc.so.6(gsignal+0x35)[0x7fb7f432b925]
/lib64/libc.so.6(abort+0x175)[0x7fb7f432d105]
/usr/sbin/mysqld[0x98459c]
/usr/sbin/mysqld[0x97ab89]
/usr/sbin/mysqld[0x97af6b]
/usr/sbin/mysqld[0x961578]
/usr/sbin/mysqld[0x9677fc]
/usr/sbin/mysqld[0xa384df]
/usr/sbin/mysqld[0xa79b28]
/usr/sbin/mysqld[0x9dd850]
/lib64/libpthread.so.0(+0x79d1)[0x7fb7f5cc29d1]
/lib64/libc.so.6(clone+0x6d)[0x7fb7f43e1b6d]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
140321 12:58:48 mysqld_safe mysqld from pid file /opt/mysql/dbprod/tools-q0.pid ended

[monty solomon (monty+launchpad)]

Crash recovery worked correctly after downgrading to 5.5

[Valerii Kravchuk (valerii-kravchuk)]

Can you try to resolve stack trace from this assertion failure on your system?

See also similar assertion among others in (hardly closely related) http://bugs.mysql.com/bug.php?id=71515.

[monty solomon (monty+launchpad)]

0x8ca6d5 my_print_stacktrace + 53
0x645024 handle_fatal_signal + 1220
0x7f6f82ec1710 _end + -2118533312
0x7f6f81522925 _end + -2145398443
0x7f6f81524105 _end + -2145392331
0x990aac page_find_rec_max_not_deleted(unsigned char const*) + 30988
0x987099 void ut_list_remove<ut_list_base<os_event>, os_event>(ut_list_base<os_event>&, os_event&, unsigned long) + 24105
0x98747b void ut_list_remove<ut_list_base<os_event>, os_event>(ut_list_base<os_event>&, os_event&, unsigned long) + 25099
0x96da88 recv_dblwr_t::add(unsigned char*) + 2936
0x973d0c recv_dblwr_t::find_first_page(unsigned long) + 20844
0xa449ef ib_cursor_new_trx(ib_cursor_t*, trx_t*) + 178399
0xa86038 fil_space_is_corrupt(unsigned long) + 2744
0x9e9d60 ib_counter_t<unsigned long, 1, single_indexer_t>::ib_counter_t() + 304

[monty solomon (monty+launchpad)]

Percona-Server-client-56-5.6.16-rel64.2.el6.x86_64
Percona-Server-devel-56-5.6.16-rel64.2.el6.x86_64
Percona-Server-shared-56-5.6.16-rel64.2.el6.x86_64
Percona-Server-server-56-5.6.16-rel64.2.el6.x86_64
Percona-Server-shared-compat-5.1.68-rel14.6.551.rhel6.x86_64

[monty solomon (monty+launchpad)]

Using upstream 5.6 instead of Percona Server 5.6 also crashes

mysql-community-release-el6-5.noarch
mysql-community-client-5.6.16-1.el6.x86_64
mysql-community-libs-5.6.16-1.el6.x86_64
mysql-community-server-5.6.16-1.el6.x86_64
mysql-community-libs-compat-5.6.16-1.el6.x86_64
mysql-community-common-5.6.16-1.el6.x86_64
mysql-community-devel-5.6.16-1.el6.x86_64

[monty solomon (monty+launchpad)]

I don't know if the crash using MySQL 5.6 was due to changes made by Percona Server 5.6 when it was trying to start up.

[monty solomon (monty+launchpad)]

old
5.5.35-33.0-log Percona Server (GPL), Release rel33.0, Revision 611

new
5.6.16-64.2-56-log Percona Server (GPL), Release 64.2, Revision 569

21:48:19 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/
key_buffer_size=67108864
read_buffer_size=1048576
max_used_connections=0
max_threads=502
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1614987 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8ca6d5]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x645024]
/lib64/libpthread.so.0(+0xf710)[0x7f642fa76710]
/usr/sbin/mysqld[0xa73127]
/usr/sbin/mysqld[0xa77df9]
/usr/sbin/mysqld[0x9edf0f]
/usr/sbin/mysqld[0x92aee4]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x588a28]
/usr/sbin/mysqld[0x6d45b1]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb5e)[0x6d818e]
/usr/sbin/mysqld[0x57b5b4]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x455)[0x580635]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f642e0c3d1d]
/usr/sbin/mysqld[0x571b31]

[monty solomon (monty+launchpad)]

0x8ca6d5 my_print_stacktrace + 53
0x645024 handle_fatal_signal + 1220
0x7f642fa76710 _end + 779423552
0xa73127 void ut_list_remove<ut_list_base<dict_table_t>, dict_table_t>(ut_list_base<dict_table_t>&, dict_table_t&, unsigned long) + 10423
0xa77df9 void ut_list_remove<ut_list_base<dict_table_t>, dict_table_t>(ut_list_base<dict_table_t>&, dict_table_t&, unsigned long) + 30089
0x9edf0f ib_counter_t<unsigned long, 1, single_indexer_t>::ib_counter_t() + 17119
0x92aee4 innobase_get_trx() + 12452
0x588a28 ha_initialize_handlerton(st_plugin_int*) + 72
0x6d45b1 lock_plugin_data() + 257
0x6d818e plugin_init(int*, char**, int) + 2910
0x57b5b4 unireg_abort + 2628
0x580635 mysqld_main(int, char**) + 1109
0x7f642e0c3d1d _end + 752477517
0x571b31 _start + 41

[Alexey Kopytov (akopytov)]

See also bug #1255476. I'm pretty sure it is the same underlying problem (i.e. some backward incompatibility wrt crash recovery + compressed tables introduced in MySQL 5.6).

[Alexey Kopytov (akopytov)]

See also http://bugs.mysql.com/bug.php?id=71515

[Valerii Kravchuk (valerii-kravchuk)]

Check upstream http://bugs.mysql.com/bug.php?id=72851. I think this is the root cause.

[Laurynas Biveinis (laurynas-biveinis)]

Marking as a duplicate of bug 1255476.