Assertion failure ut_a(page_zip_verify_checksum(frame, zip_size)) still happens when it should not

[Valerii Kravchuk]

On Percona Server 5.6.22 (where "fix" for upstream bug http://bugs.mysql.com/bug.php?id=73689 should be included included) we still get assertion failures like this:

2015-06-20 12:41:32 7f39737fe700 InnoDB: Assertion failure in thread 139884727625472 in file buf0flu.cc line 940
InnoDB: Failing assertion: page_zip_verify_checksum(frame, zip_size)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:41:32 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=1658
max_threads=8022
thread_count=384
connection_count=384
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 82296044 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x90aa6c]
/usr/sbin/mysqld(handle_fatal_signal+0x352)[0x6644b2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f5287024cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f52866900d5]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f528669383b]
/usr/sbin/mysqld[0xa663b9]
/usr/sbin/mysqld[0xa66b32]
/usr/sbin/mysqld[0xa679aa]
/usr/sbin/mysqld[0xa68795]
/usr/sbin/mysqld[0xa6a47f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f528701ce9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f528674e2ed]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
150620 12:41:41 mysqld_safe Number of processes running now: 0

Moreover, code in current Percona Server 5.6.24 is the same as mentioned in last comments in the upstream http://bugs.mysql.com/bug.php?id=73689:

937 case BUF_BLOCK_ZIP_DIRTY:
938 frame = bpage->zip.data;
939
940 ut_a(page_zip_verify_checksum(frame, zip_size));
941
942 mach_write_to_8(frame + FIL_PAGE_LSN,
943 bpage->newest_modification);
944 memset(frame + FIL_PAGE_FILE_FLUSH_LSN, 0, 8);
945 break;

So, LSN is NOT written to the page before checking that it's corrupted.

I suspect that there is Oracle internal bug reported for upstream MySQL that is NOT yet fixed in 5.6.25. So, we probably have to fix this for Percona Server 5.6.25, as we do not know when/if/how Oracle is going to fix this problem entirely.

[Laurynas Biveinis (laurynas-biveinis)]

https://github.com/percona/percona-server/pull/98

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-909