InnoDB: Assertion failure in thread <nr> in file fil0fil.c line 844

Bug #1246916 reported by George Ormond Lorch III
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Status tracked in 5.7
5.1
Won't Fix
Undecided
Unassigned
5.5
New
Undecided
Unassigned
5.6
New
Undecided
Unassigned
5.7
New
Undecided
Unassigned

Bug Description

Testing crash safe slave on PS 5.5.33-29.3

Configure master-slave with slave having following options:
innodb_file_format=barracuda
innodb_file_per_table=1
innodb_recovery_update_relay_log=1
innodb_flush_log_at_trx_commit=0
log-bin=a
log-slave-updates=TRUE

Start slave and sync to master position.
Start RQG against master.
kill -9 slave
Restart slave:
131031 15:27:28 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: Warning: innodb_overwrite_relay_log_info is enabled. Updates by other storage engines may not be synchronized.
InnoDB: relay-log.info is detected.
InnoDB: relay log: position 1579421, file name ./mysql-relay-bin.000002
InnoDB: master log: position 1579484, file name a.000003
131031 15:27:28 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
131031 15:27:28 InnoDB: The InnoDB memory heap is disabled
131031 15:27:28 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131031 15:27:28 InnoDB: Compressed tables use zlib 1.2.3
131031 15:27:28 InnoDB: Using Linux native AIO
131031 15:27:28 InnoDB: Initializing buffer pool, size = 128.0M
131031 15:27:28 InnoDB: Completed initialization of buffer pool
131031 15:27:28 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 24664943
131031 15:27:28 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Unable to lock ./rqg1/table100_innodb_compressed.ibd, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
131031 15:27:28 InnoDB: Assertion failure in thread 140293712815904 in file fil0fil.c line 844
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:27:28 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 342996 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x35)[0x7ee635]
mysqld(handle_fatal_signal+0x314)[0x6b55e8]
/lib64/libpthread.so.0[0x391b00f500]
/lib64/libc.so.6(gsignal+0x35)[0x391a8328a5]
/lib64/libc.so.6(abort+0x175)[0x391a834085]
mysqld[0x8f1e71]
mysqld[0x8f2070]
mysqld[0x8f24ef]
mysqld[0x8f252c]
mysqld[0x8727c3]
mysqld[0x937a58]
mysqld[0x93bf0a]
mysqld[0x93d3a3]
mysqld[0x8579a7]
mysqld[0x80498f]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0xa4)[0x6b8c7e]
mysqld[0x5acf80]
mysqld(_Z11plugin_initPiPPci+0xac1)[0x5b07df]
mysqld[0x51fc9b]
mysqld(_Z11mysqld_mainiPPc+0x609)[0x520a2c]
mysqld(main+0x9)[0x515d9d]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x391a81ecdd]
mysqld[0x515cd9]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file

Tags: i38419
Revision history for this message
George Ormond Lorch III (gl-az) wrote :

thread apply all bt

Revision history for this message
George Ormond Lorch III (gl-az) wrote :

Running same test overnight hit this same thing 3 more times out of 50-60 cycles.

Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

If this still happens with recent version and this report is supposed to be confirmed by somebody else, I think RQG grammar file to run is missing...

Changed in percona-server:
status: New → Incomplete
Revision history for this message
George Ormond Lorch III (gl-az) wrote :

This used the standard percona grammar file:
RQG=<path to rqg>
PORT=<master port>
perl ${RQG}/gentest.pl --dsn=dbi:mysql:host=127.0.0.1:port=${PORT}:user=root:database=rqg1 --gendata=${RQG}/conf/percona_qa/percona_qa.zz --grammar=${RQG}/conf/percona_qa/percona_qa.yy

I can grab the current trunks and retry but I think it will still exist as this was found on PS trunk just a week or two ago and AFAIK no major new changes have been made since.

Changed in percona-server:
status: Incomplete → New
Revision history for this message
Przemek (pmalkowski) wrote :

I have seen very similar crash in a case where MySQL data directory was located on a NFS share. After the MySQL server host was rebooted (not clean reboot), MySQL could not gain write permissions (error 11) to some files despite the fact that from OS level all seemed right. Only copying each file out and back in resolved the access/locking problem.
This is example crash:

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Unable to lock ./db/table#P#p1#SP#p1sp0.ibd, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
140110 13:22:45 InnoDB: Assertion failure in thread 140545018697664 in file fil0fil.c line 844
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.

Happened on Percona Server 5.5.34-32.0-log

Przemek (pmalkowski)
tags: added: i38419
Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

See also upstream bug http://bugs.mysql.com/bug.php?id=65276 with unclear status (and on Windows).

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3060

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.