Unclean shutdown corrupted Aria table blocking startup

[Dreas van Donselaar]

On 3 different servers now MariaDB failed to start after a power failure:
====
Jul 22 15:54:38 filt01 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Jul 22 15:54:38 filt01 mysqld: 110722 15:54:38 [Warning] '--log_slow_queries' is deprecated and will be removed in a future release. Please use ''--slow_query_log'/'--log-slow-file'' instead.
Jul 22 15:54:38 filt01 mysqld: 110722 15:54:38 [Warning] 'master-host' is deprecated and will be removed in a future release. Please use ''CHANGE MASTER'' instead.
Jul 22 15:54:38 filt01 mysqld: 110722 15:54:38 [ERROR] mysqld got signal 11 ;
Jul 22 15:54:38 filt01 mysqld: This could be because you hit a bug. It is also possible that this binary
Jul 22 15:54:38 filt01 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Jul 22 15:54:38 filt01 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul 22 15:54:38 filt01 mysqld: We will try our best to scrape up some info that will hopefully help diagnose
Jul 22 15:54:38 filt01 mysqld: the problem, but since we have already crashed, something is definitely wrong
Jul 22 15:54:38 filt01 mysqld: and this may fail.
Jul 22 15:54:38 filt01 mysqld:
Jul 22 15:54:38 filt01 mysqld: key_buffer_size=268435456
Jul 22 15:54:38 filt01 mysqld: read_buffer_size=131072
Jul 22 15:54:38 filt01 mysqld: max_used_connections=0
Jul 22 15:54:38 filt01 mysqld: max_threads=502
Jul 22 15:54:38 filt01 mysqld: threads_connected=0
Jul 22 15:54:38 filt01 mysqld: It is possible that mysqld could use up to
Jul 22 15:54:38 filt01 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1359776 K
Jul 22 15:54:38 filt01 mysqld: bytes of memory
Jul 22 15:54:38 filt01 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Jul 22 15:54:38 filt01 mysqld:
Jul 22 15:54:38 filt01 mysqld: Thread pointer: 0x0
Jul 22 15:54:38 filt01 mysqld: Attempting backtrace. You can use the following information to find out
Jul 22 15:54:38 filt01 mysqld: where mysqld died. If you see no messages after this, something went
Jul 22 15:54:38 filt01 mysqld: terribly wrong...
Jul 22 15:54:38 filt01 mysqld: stack_bottom = (nil) thread_stack 0x40000
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7f2c65bc3119]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(handle_segfault+0x439)[0x7f2c6580e709]
Jul 22 15:54:38 filt01 mysqld: /lib/libpthread.so.0[0x7f2c63993a80]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(trnman_get_min_safe_trid+0x23)[0x7f2c659ed103]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(maria_open+0x2595)[0x7f2c65a0a265]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c65a1f388]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c65a1fccc]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(maria_apply_log+0x445)[0x7f2c65a241f5]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(maria_recovery_from_log+0x50)[0x7f2c65a248d0]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c659dd821]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x38)[0x7f2c658f59e8]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c65979c4c]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x839)[0x7f2c6597c0b9]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c65810637]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld(main+0x1f0)[0x7f2c658111e0]
Jul 22 15:54:38 filt01 mysqld: /lib/libc.so.6(__libc_start_main+0xe6)[0x7f2c636501a6]
Jul 22 15:54:38 filt01 mysqld: /usr/sbin/mysqld[0x7f2c65749ae9]
Jul 22 15:54:38 filt01 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jul 22 15:54:38 filt01 mysqld: information that should help you find out what is causing the crash.
Jul 22 15:54:38 filt01 mysqld: Writing a core file
Jul 22 15:54:38 filt01 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
====

We have 1 table in Aria format. As soon as we move out the Aria table files, MariaDB starts fine again.

I've uploaded a copy of such a corrupt Aria table to the MariaDB FTP server (private folder).

[Michael Widenius (monty)]

I am uploading the test case just now.
Note that to fix a recovery problem, I would need the following files:
- The table files (.MAI, .MAD and .frm)
- The aria_log_control file
- The aria log files (aria_log.*)
- Optionally the aria_recovery.trace file

I will come back to you if the tar file doesn't contain all of the above files.

[Dreas van Donselaar (dreas-9)]

Hi!

On 07/26/2011 06:33 PM, Michael Widenius wrote:
> I am uploading the test case just now.
> Note that to fix a recovery problem, I would need the following files:
> - The table files (.MAI, .MAD and .frm)
> - The aria_log_control file
> - The aria log files (aria_log.*)
> - Optionally the aria_recovery.trace file
>
> I will come back to you if the tar file doesn't contain all of the above
> files.
It does not. I only included the table files. I'm afraid I have to wait
for another incident to include the other files.

Greets,

Dreas

[Dreas van Donselaar (dreas-9)]

We had a similar crash now after simply issueing on a server:
===
mysql -e "stop slave; set global sql_slave_skip_counter=1; start slave";
===

I'll upload a new copy including aria_log_control / aria_log.* of the new data to the FTP.

[Dreas van Donselaar (dreas-9)]

I just moved out the table files, started MySQL, moved the table files back in, and ran a repair:
===
server:/var/lib/mysql/spambayes$ mysqlcheck --auto-repair spambayes
spambayes.global_tokens
warning : Table is marked as crashed
warning : 1 client is using or hasn't closed the table properly
error : Page 126165: Row 161 is crashed
error : Corrupt
spambayes.hashed OK
spambayes.local_tokens OK

Repairing tables
spambayes.global_tokens
warning : Number of rows changed from 21682848 to 21682752
status : OK
server:/var/lib/mysql/spambayes$
===
Not sure if this provides additional useful information.

[Michael Widenius (monty)]

I was able to reproduce and fix this.
Fix is pushed into 5.2

[Dreas van Donselaar (dreas-9)]

This bug is not yet fixed. We just hit it again running 5.2.8. Uploading tgz with required files to the FTP server now.

[Dreas van Donselaar (dreas-9)]

This issue is still occurring (with 5.2.9). Uploading the requested files to the FTP server now.

[Dreas van Donselaar (dreas-9)]

Switching back to in progress because the issue is still occurring.

[Michael Widenius (monty)]

I think this is fixed in 5.2.11. If not ,please reopen the bug!