Archive table corruption crashing MariaDB signal 11

[Dreas van Donselaar]

We had a MariaDB 5.2.10 server that was continuously crashing:
{{{
Jan 17 16:17:07 server1 mysqld: 120117 16:17:07 [ERROR] mysqld got signal 11 ;
Jan 17 16:17:07 server1 mysqld: This could be because you hit a bug. It is also possible that this binary
Jan 17 16:17:07 server1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jan 17 16:17:07 server1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jan 17 16:17:07 server1 mysqld: We will try our best to scrape up some info that will hopefully help diagnose
Jan 17 16:17:07 server1 mysqld: the problem, but since we have already crashed, something is definitely wrong
Jan 17 16:17:07 server1 mysqld: and this may fail.
Jan 17 16:17:07 server1 mysqld:
Jan 17 16:17:07 server1 mysqld: key_buffer_size=0
Jan 17 16:17:07 server1 mysqld: read_buffer_size=131072
Jan 17 16:17:07 server1 mysqld: max_used_connections=17
Jan 17 16:17:07 server1 mysqld: max_threads=502
Jan 17 16:17:07 server1 mysqld: threads_connected=6
Jan 17 16:17:07 server1 mysqld: It is possible that mysqld could use up to
Jan 17 16:17:07 server1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1098540 K
Jan 17 16:17:07 server1 mysqld: bytes of memory
Jan 17 16:17:07 server1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jan 17 16:17:07 server1 mysqld:
Jan 17 16:17:07 server1 mysqld: Thread pointer: 0x7fd61f517b70
Jan 17 16:17:07 server1 mysqld: Attempting backtrace. You can use the following information to find out
Jan 17 16:17:07 server1 mysqld: where mysqld died. If you see no messages after this, something went
Jan 17 16:17:07 server1 mysqld: terribly wrong...
Jan 17 16:17:07 server1 mysqld: stack_bottom = 0x2020202020202020 thread_stack 0x40000
Jan 17 16:17:07 server1 mysqld: *** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption: 0x00007fd62147d000 ***
}}}
Initially we thought this would be caused by hardware issues, but it's a repair of an archive table that triggers this. I've uploaded the table causing this crash to the FTP server.

[Elena Stepanova (elenst)]

Hi,

You are saying that it's crashing on 5.2.10. Do you have a server where this table works okay?
From what I see, any action on this table also causes crashes on MySQL-5.1.60, MySQL-5.6.4, MariaDB-5.2.8.

When did the crashes start? On which server version was this table updated for the last time?

[Dreas van Donselaar (dreas-9)]

Hi,

> You are saying that it's crashing on 5.2.10. Do you have a server where this table works okay?
No, it seems to fail everywhere (fortunately the data inside the table is not important to us though).

> When did the crashes start? On which server version was this table updated for the last time?
The crash started after an unclean shutdown of the machine. It was last updated in 5.2.10.

[Elena Stepanova (elenst)]

So, as said before, any action on the provided table causes a server crash. Apparently, the table got corrupted due to the unclean shutdown as Dreas mentioned in his comment, but hopefully there is still a way to avoid a crash and report a table corruption instead.

[Dreas van Donselaar (dreas-9)]

This is reoccurring on one of our servers. Including backtrace output:
===
mysqlcheck: Got error: 2013: Lost connection to MySQL server during query when executing 'CHECK TABLE ... FAST'
root@mysql2 /var/lib/mysql # *** glibc detected *** /usr/sbin/mysqld: corrupted double-linked list: 0x00007f85c1bb3530 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7f86d481dbd6]
/lib/libc.so.6(+0x73608)[0x7f86d481f608]
*** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption: 0x00007ff00189f440 ***
======= Backtrace: =========
/lib/libc.so.6(+0x71bd6)[0x7feffc5a7bd6]
/lib/libc.so.6(+0x74c6d)[0x7feffc5aac6d]
/lib/libc.so.6(__libc_malloc+0x70)[0x7feffc5aca30]
/lib/libc.so.6(__backtrace_symbols+0x11f)[0x7feffc61a28f]
/usr/sbin/mysqld(my_print_stacktrace+0x5f)[0x7feffee1868f]
/usr/sbin/mysqld(handle_segfault+0x405)[0x7feffea53f05]
/lib/libpthread.so.0(+0xeff0)[0x7feffd044ff0]
/lib/libc.so.6(memset+0xacf)[0x7feffc5b4faf]
/usr/sbin/mysqld(_ZN12Field_string6unpackEPhPKhjb+0x86)[0x7feffea2d9e6]
/usr/sbin/mysqld(_ZN10ha_archive10unpack_rowEP11azio_streamPh+0x187)[0x7feffec1ca07]
/usr/sbin/mysqld(_ZN10ha_archive8optimizeEP3THDP15st_ha_check_opt+0x1c7)[0x7feffec1db87]
/usr/sbin/mysqld(_ZN10ha_archive6repairEP3THDP15st_ha_check_opt+0x15)[0x7feffec1be45]
/usr/sbin/mysqld(_ZN10ha_archive16check_and_repairEP3THD+0x38)[0x7feffec1bfa8]
/usr/sbin/mysqld(+0x3aa729)[0x7feffea9f729]
/usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootPbj+0x808)[0x7feffeaa2198]
/usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjj+0x5f7)[0x7feffeaa2b27]
/usr/sbin/mysqld(_Z28open_and_lock_tables_derivedP3THDP10TABLE_LISTb+0x67)[0x7feffeaa4b57]
/usr/sbin/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0xd8)[0x7feffeadb518]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xaf8)[0x7feffea61eb8]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x2c1)[0x7feffea66eb1]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xd0b)[0x7feffea6801b]
/usr/sbin/mysqld(_Z10do_commandP3THD+0xf4)[0x7feffea68b34]
/usr/sbin/mysqld(handle_one_connection+0x14b)[0x7feffea5aebb]
/lib/libpthread.so.0(+0x68ca)[0x7feffd03c8ca]
/lib/libc.so.6(clone+0x6d)[0x7feffc60586d]
======= Memory map: ========
7feec4000000-7feec52b6000 rw-p 00000000 00:00 0
7feec52b6000-7feec8000000 ---p 00000000 00:00 0
7feecb75e000-7feecb75f000 ---p 00000000 00:00 0
7feecb75f000-7feecb79f000 rw-p 00000000 00:00 0
7feecb79f000-7feecb7a0000 ---p 00000000 00:00 0
7feecb7a0000-7feecb7e0000 rw-p 00000000 00:00 0
7feecb7e0000-7feecb7e1000 ---p 00000000 00:00 0
7feecb7e1000-7feecb821000 rw-p 00000000 00:00 0
7feecb821000-7feecb822000 ---p 00000000 00:00 0
7feecb822000-7feecb862000 rw-p 00000000 00:00 0
7feecb862000-7feecb863000 ---p 00000000 00:00 0
7feecb863000-7feecb8a3000 rw-p 00000000 00:00 0
7feecb8a3000-7feecb8a4000 ---p 00000000 00:00 0
7feecb8a4000-7feecb8e4000 rw-p 00000000 00:00 0
7feecb8e4000-7feecb8e5000 ---p 00000000 00:00 0
7feecb8e5000-7feecb925000 rw-p 00000000 00:00 0
7feecb925000-7feecb926000 ---p 00000000 00:00 0
7feecb926000-7feecb966000 rw-p 00000000 00:00 0
7feecb966000-7feecb967000 ---p 00000000 00:00 0
7feecb967000-7feecb9a7000 ...