innodb_force_recovery is more efficient with InnoDB plugin than XtraDB on certain cases

[Sergei Golubchik]

Originally reported as https://mariadb.atlassian.net/browse/MDEV-4777
===================================
It is the second time that i have a serious power shortage on a server with a RAID card with wright through cache enabled and that has its battery depleted (the RAID card flushes its cache to disk not on a FIFO manner) with InnoDB tables on a MariaDB 5.5 ending up badly corrupted and i encounter this very issue.

These tables are write intensive as they store monitoring data.

The only way to start the server is to set innodb_force_recovery = 6 and to disable XtraDB and load InnoDB plugin instead.

Setting the innodb_force_recovery to any parameter with XtraDB will end up on either the server crashing at start (with values from 1 to 5 in my case) either with this messages looping in the error log while stays still unavailable : InnoDB: Waiting for the background threads to start

Here are the kind of errors shown on the error log when starting with innodb_force_recovery = 6 and InnoDB plugin :

130711 3:09:36 InnoDB: error: space object of table 'zabbix/acknowledges',
InnoDB: space id 10 did not exist in memory. Retrying an open.
130711 3:09:36 InnoDB: Warning: allocated tablespace 10, old maximum was 0
130711 3:09:36 InnoDB: error: space object of table 'zabbix/actions',
InnoDB: space id 11 did not exist in memory. Retrying an open.
130711 3:09:36 InnoDB: error: space object of table 'zabbix/alerts',

Accessing certain rows on certain tables will crash MariaDB but at least the other tables can be dumped completely and these tables can also be partially dumped using a LIMIT clause.
===================================

As further comments indicate, a possible reason could be innodb_corrupt_table_action variable.

[Valerii Kravchuk (valerii-kravchuk)]

It would be nice to get my.cnf content from the environment affected.

[Jean Weisbuch (m-jea0-p)]

Hi,

Here is the my.cnf file used on this particular server :

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmpfs
skip-external-locking
bind-address = 127.0.0.1
key_buffer = 16M
max_allowed_packet = 512M
thread_cache_size = 10
myisam-recover = BACKUP
max_connections = 60
table_open_cache = 400
table_definition_cache = 400
query_cache_type = 2
query_cache_limit = 1M
query_cache_size = 16M
query_cache_strip_comments = 1
log_error = /var/log/mysql/error.log
slow_query_log = 0
slow_query_log_file = /var/log/mysql/mysql-slow.log
long_query_time = 4
expire_logs_days = 10
max_binlog_size = 100M
local-infile = 0
open_files_limit = 65535
skip-name-resolve
join_buffer_size = 2M
read_rnd_buffer_size = 1M
tmp_table_size = 64M
max_heap_table_size = 64M
max_connect_errors = 1000
bulk_insert_buffer_size = 128M
query_prealloc_size = 16K
myisam_sort_buffer_size = 128M
concurrent_insert = 2
default_storage_engine = InnoDB
innodb_file_format = Barracuda
innodb_buffer_pool_size = 1280M
innodb_log_file_size = 128M
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
join_cache_level = 2
key_cache_segments = 2
[mysqldump]
quick
quote-names
max_allowed_packet = 512M
[mysql]
[isamchk]
key_buffer = 512M

And in order to access the datas i had to add :

innodb_force_recovery = 6
ignore-builtin-innodb
plugin-load=ha_innodb.so
innodb

I didnt knew about the innodb_corrupt_table_action variable at the moment, a mention about it on the recovery error messages could be interresting as its a XtraDB only variable.

[Valerii Kravchuk (valerii-kravchuk)]

It seems a duplicate of https://bugs.launchpad.net/percona-server/+bug/923820

That's because in Percona Server 5.5 innodb_purge_threads=1 by default (see http://www.percona.com/doc/percona-server/5.5/scalability/innodb_io_55.html#innodb_purge_threads), and with this value forced recovery just does not work. So, until that other bug is fixed, you have to set this variable to 0 during forced recovery.

[Launchpad Janitor (janitor)]

[Expired for Percona Server because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3003