RQG: Table corruption after Maria engine recovery - "Wrong data in bitmap"

[Philip Stoev]

After running a DML workload for 20 seconds, followed by a kill -9 and a (successful) recovery, CHECK TABLE EXTENDED on a Maria table reported the following error:

            'Page 2: Wrong data in bitmap. Page_type: 1 empty_space: 6321 Bitmap-bits: 3'

The datadir will be uploaded shortly.

[Philip Stoev (philip-stoev)]

This issues seems to require =--maria-block-size=16K to reproduce. To reproduce with the RQG:

perl runall.pl --basedir=/home/philips/bzr/5.2 --grammar=conf/engines/engine_stress.yy --gendata=conf/engines/engine_stress.zz --queries=1M --engine=Maria --mysqld=--log-output=file --reporters=ErrorLog,Backtrace,Recovery --duration=20 --threads=1 --rows=1000 --mask-level=2 --mysqld=--maria-block-size=16K --mask=25551 --queries=100000000 --mysqld=--loose-skip-innodb --mysqld=--loose-skip-pbxt

[Michael Widenius (monty)]

Fix pushed into 5.1

[Philip Stoev (pstoev-askmonty)]

This bug is still repeatable on the latest 5.1 with the patch included.

I am attaching a new tablespace that produces the following error:

          [
            'test.table1000_maria_int_autoinc',
            'check',
            'error',
            'Page 41: Wrong data in bitmap. Page_type: 0 full: 1 empty_space: 811 Bitmap-bits: 4'
          ],

[Philip Stoev (pstoev-askmonty)]

Here is an example RQG command line

/home/buildbot/randgen/runall.pl \
--grammar=conf/engines/engine_stress.yy \
--gendata=conf/engines/engine_stress.zz \
--queries=1M \
--engine=Maria \
--mysqld=--log-output=file \
--mysqld=--log-bin \
--reporters=ErrorLog,Backtrace,Recovery,Shutdown \
--duration=240 \
--threads=5 \
--rows=10000 \
--mask-level=2 \
--mysqld=--maria-repair-threads=2 \
--mysqld=--loose-maria-group-commit=soft \
--mysqld=--loose-maria_group_commit_interval=100000000 \
--mysqld=--maria-checkpoint-interval=1 \
--mysqld=--maria-block-size=16K \
--mask=64891 \
--queries=100000000 \
--basedir=. \
--seed=1281526887

[Michael Widenius (monty)]

To repeat:
- get above file var-bug602169-2.zip
Exceute the following:

cd mysql-test
unzip var-bug602169-2.zip
rm -rf var
mv vardir8 var
MTR_VERSION=1 perl mysql-test-run.pl --start-dirty --mysqld=--maria 1st --mysqld=--maria_block_size=1024
maria_chk ./storage/maria/maria_chk -s var/master-data/test/table1000_maria_int_autoinc

You will get the output:

maria_chk: error: Page 41: Wrong data in bitmap. Page_type: 1 full: 0 empty_space: 811 Bitmap-bits: 4

[Michael Widenius (monty)]

To fix this, I need the exact command to repeat this new failure so that I can repeat this.
(I will probably need to generate a debug trace of mysqld to track this bug down)

Note to Philip: The problem is probably a timing issue with maria_checkpoints, so if you execute checkpoints more often it may be easier to repeat this.

[Philip Stoev (pstoev-askmonty)]

RQG command line that reproduced this issue with checkpoint interval = 0

/home/buildbot/randgen/runall.pl \
 --queries=1M \
 --engine=Maria \
 --mysqld=--log-output=file \
 --reporters=ErrorLog,Backtrace,Recovery,Shutdown \
 --duration=240 \
 --threads=20 \
 --rows=10 \
 --mask-level=1 \
 --mysqld=--loose-maria-group-commit=hard \
 --mysqld=--loose-maria_group_commit_interval=10 \
 --mysqld=--maria-checkpoint-interval=0 \
 --mysqld=--maria-block-size=16K \
 --grammar=conf/engines/many_indexes.yy \
 --gendata=conf/engines/many_indexes.zz \
 --mask=53801 \
 --queries=100000000 \
 --basedir=. \
 --seed=1283009930

[Philip Stoev (pstoev-askmonty)]

This command line reproduces the bug with certainty . It uses --mem and --mysqld=--maria-checkpoint-interval=0

 perl runall.pl --queries=1M --engine=Maria --mysqld=--log-output=file --reporters=ErrorLog,Backtrace,Recovery,Shutdown --duration=240 --threads=20 --rows=10 --mask-level=1 --mysqld=--loose-maria-group-commit=hard --mysqld=--loose-maria_group_commit_interval=10 --mysqld=--maria-checkpoint-interval=0 --mysqld=--maria-block-size=16K --grammar=conf/engines/many_indexes.yy --gendata=conf/engines/many_indexes.zz --mask=53801 --queries=100000000 --seed=1283009930 --basedir=/home/philips/bzr/maria-5.1 --duration=60 --mysqld=--skip-innodb --mem

Please use the following .ZZ file

$tables = {
        rows => [10]
};

$fields = {
        types => [ 'varchar(255)', 'char(255)' , 'enum', 'set', 'longtext' ],
        indexes => [ undef, 'key' ],
        charsets => [ 'UTF8', 'UCS2' ]
};

$data = {
        strings => [ 'letter', 'english', undef ],
        blobs => [ 'letter', 'english' ]
}

[Philip Stoev (pstoev-askmonty)]

Still repeatable with

revision-id: <email address hidden>
date: 2010-09-07 19:58:39 +0300
build-date: 2010-09-09 10:49:08 +0300
revno: 2924
branch-nick: 5.1

RQG command line:

# 2010-09-09T06:21:29 # /home/buildbot/randgen/runall.pl \
# 2010-09-09T06:21:29 # --queries=1M \
# 2010-09-09T06:21:29 # --engine=Maria \
# 2010-09-09T06:21:29 # --mysqld=--log-output=file \
# 2010-09-09T06:21:29 # --mysqld=--maria_log_purge_type=at_flush \
# 2010-09-09T06:21:29 # --reporters=ErrorLog,Backtrace,Recovery,Shutdown \
# 2010-09-09T06:21:29 # --duration=120 \
# 2010-09-09T06:21:29 # --threads=20 \
# 2010-09-09T06:21:29 # --rows=10000 \
# 2010-09-09T06:21:29 # --mask-level=2 \
# 2010-09-09T06:21:29 # --mysqld=--maria-repair-threads=2 \
# 2010-09-09T06:21:29 # --mysqld=--loose-maria-group-commit=soft \
# 2010-09-09T06:21:29 # --mysqld=--loose-maria_group_commit_interval=0 \
# 2010-09-09T06:21:29 # --mysqld=--maria-checkpoint-interval=1 \
# 2010-09-09T06:21:29 # --mysqld=--maria-block-size=4K \
# 2010-09-09T06:21:29 # --grammar=conf/engines/engine_stress.yy \
# 2010-09-09T06:21:29 # --gendata=conf/engines/engine_stress.zz \
# 2010-09-09T06:21:29 # --mask=6680 \
# 2010-09-09T06:21:29 # --queries=100000000 \
# 2010-09-09T06:21:29 # --mask=6680 \
# 2010-09-09T06:21:29 # --basedir=. \
# 2010-09-09T06:21:29 # --seed=time \
# 2010-09-09T06:21:29 # --vardir=/home/buildbot/rqg-buildbot-slave/rqg-weekend/vardir-maria_recovery_weekend/current