xb_incremental_bitmap_misc.sh fails in debug xtradb56 builds | missing bitmap data at the interval start fails " bitmap_files->files[0].start_lsn == first_file_start_lsn" assertion

[Alexey Kopytov]

After updating PS version used by the test suite from PS 5.6.10 to PS 5.6.11 (i.e. the one where page tracking is available), xb_incremental_bitmap_misc.sh fails in debug xtradb56 builds as follows:

incremental backup from 9000 is enabled.
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mnt/workspace/percona-xtrabackup-2.1-param/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/xtradb56/test/var/w1/var1/data
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
>> log scanned up to (1634133)
InnoDB: Allocated tablespace 6, old maximum was 0
2013-07-22 12:10:05 2acfb2ea2580 InnoDB: Assertion failure in thread 47071548286336 in file changed_page_bitmap.cc line 404
InnoDB: Failing assertion: bitmap_files->files[0].start_lsn == first_file_start_lsn
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
inc/common.sh: line 70: 6377 Aborted (core dumped) "$@"
2013-07-22 12:10:05: run.sh: ===> xtrabackup_56 failed with exit code 134
------------

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/386/BUILD_TYPE=debug,Host=centos5-64,xtrabackuptarget=xtradb56/testReport/junit/%28root%29/t_xb_incremental_bitmap_misc/sh/

[Laurynas Biveinis (laurynas-biveinis)]

The test fails in the debug code of changed_page_bitmap.cc, at the last test step, where it attempts to perform a bitmap backup (LSNs 9000..infinity)) with bitmap data missing for the start of interval.

With PS 5.6.11 the bitmap files are as follows:
     /home/laurynas/percona/src/bug1204075/test/var/w1/var1/data/ib_modified_log_4_1633805.xdb
     /home/laurynas/percona/src/bug1204075/test/var/w1/var1/data/ib_modified_log_5_1634123.xdb

With PS 5.5 the bitmap files are as follows:
     /home/laurynas/percona/src/bug1204075/test/var/w1/var1/data/ib_modified_log_4_1602083.xdb
     /home/laurynas/percona/src/bug1204075/test/var/w1/var1/data/ib_modified_log_5_1602404.xdb

I.e. identical for the purposes of the test. Given that xtradb51 and xtradb55 debug configurations are disabled in Jenkins and at least the xtradb55 one does not even build, I believe this bug has been from day one and surfaced on the first actually enabled debug xtradb configuration.

[Laurynas Biveinis (laurynas-biveinis)]

The release builds diagnose the missing LSN data at the interval start at xb_page_bitmap_init(), after
log_online_setup_bitmap_file_range() completes:

xtrabackup: warning: changed page data missing for LSNs between 9000 and 1633805
xtrabackup: using the full scan for incremental backup

But the debug build will attempt to assert that such situation is impossible:

ut_ad(bitmap_files->files[0].start_lsn == first_file_start_lsn);

If no bitmap file with LSN <= range start LSN, first_file_start_lsn will be LSN_MAX, and files[0].start_lsn will be the lowest LSN of the actual files found (1633805 in 5.6 example). Given that the missing data is diagnosed later, and log_online_setup_bitmap_file_range() is not the correct place to report missing data errors (xb_page_bitmap_init() is), the fix is to remove this assert as overzealous.

This bug is shared with PS I_S query code, where a corresponding testcase needs to be written and the fix ported too.

[Roel Van de Paar (roel11)]

Agreed w/ Laurynas on 56qual, so proposing high for re-triage on 5.6 only

[Laurynas Biveinis (laurynas-biveinis)]

The PS fix is proposed at bug 1191580. I failed to remember this bug while fixing that bug. See comment #24 there:

"Thus now there are two different fixes for this issue in the source shared between XB and PS, one merged to XB in 1204075, and one proposed for PS here. I am not sure how to best handle this situation. The current fix seems to be better and suitable for XB eventually as well. Thus if nobody objects I will remove XB from this bug and PS from bug 1204075."

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-887