WaffleGrid Crashes innodb if read-ahead enabled

Bug #382777 reported by MadMatt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
WaffleGrid
Fix Committed
High
MadMatt

Bug Description

Memcached set (via disk read): Block: 2fd5:4:f3f : Thread : 1200658768 time:117 settime: 1243950942 : 794113
Memcached miss: Block: 2fd5:9:3 : Thread : 1200658768 time:248
Memcached miss: Block: 2fd5:9:1 : Thread : 1200658768 time:44
Memcached set (via disk read): Block: 2fd5:9:1 : Thread : 1200658768 time:459 settime: 1243950946 : 155809
Memcached set (via disk read): Block: 2fd5:9:3 : Thread : 1200658768 time:995 settime: 1243950946 : 155985
Memcached miss: Block: 2fd5:9:2 : Thread : 1200658768 time:45
Memcached set (via disk read): Block: 2fd5:9:2 : Thread : 1200658768 time:440 settime: 1243950946 : 156495
Memcached miss: Block: 2fd5:0:0 : Thread : 1200658768 time:0
Memcached miss: Block: 2fd5:0:2 : Thread : 1200658768 time:0
Memcached get: Block: 2fd5:7:b3 : Thread : 1200658768 time:135
InnoDB: Page directory corruption: infimum not pointed to
090602 9:55:46 InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000

...

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ;InnoDB: End of page dump
090602 9:55:46 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
InnoDB: Page directory corruption: supremum not pointed to
090602 9:55:46 InnoDB: Page dump in ascii and hex (16384 bytes):
 len 16384; hex 000000
090602 9:55:46 InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
InnoDB: Page number (if stored to page already) 0,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be a freshly allocated page
090602 9:55:46InnoDB: Error: trying to access a stray pointer 0x80007f8f58023ff8
InnoDB: buf pool start is at 0x7f8f482dc000, end at 0x7f8f782dc000
InnoDB: Probable reason is database corruption or memory
InnoDB: corruption. If this happens in an InnoDB database recovery, see
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: how to force recovery.
090602 9:55:46 InnoDB: Assertion failure in thread 1200658768 in file ../../storage/innobase/include/buf0buf.ic line 264
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
[0x7a4fb5]
[0x4ce335]
[0x656ac0]
[0x891525]
[0x8561c3]
[0x6dbad4]
[0x67f2ab]
[0x6fc5a3]
[0x67167c]
[0x5bd85a]
[0x532075]
[0x539e1e]
[0x53f4b9]
[0x54109b]
[0x54461f]
[0x54a0c1]
[0x4dbdd8]
[0x4dfb82]
[0x4e45e7]
[0x4e567f]
[0x4e5e2e]
[0x4d84bd]
[0x65311d]
[0x879669]
090602 9:55:46 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=67108864
read_buffer_size=262144
max_used_connections=31
max_threads=151
threads_connected=31
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 144381 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x3a51920
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x47908f70 thread_stack 0x40000
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x3b73d80 = SELECT c_discount, c_last, c_credit
FROM customer
WHERE c_w_id = 1
  AND c_d_id = 2
  AND c_id = 608
thd->thread_id=10
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
090602 09:55:46 mysqld_safe Number of processes running now: 0
090602 09:55:46 mysqld_safe mysqld restarted
InnoDB: The InnoDB memory heap has been disabled.
InnoDB: Mutex and rw_lock use atomics.
InnoDB: ios_per_array 256 read threads 4 write threads 4
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
090602 9:55:46 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...

Parms:

# new parms
#innodb_adaptive_checkpoint=1
#innodb_read_ahead=0
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=800
#innodb_enable_seq_readahead=0
#innodb_enable_rnd_readahead=0
#skip-innodb-doublewrite

Related branches

Revision history for this message
MadMatt (yonkovim) wrote :

This has crashed 2x in a row.

Revision history for this message
MadMatt (yonkovim) wrote :

With
# new parms
#innodb_adaptive_checkpoint=1
#innodb_read_ahead=0
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_io_capacity=800
innodb_enable_seq_readahead=0
innodb_enable_rnd_readahead=0
#skip-innodb-doublewrite

We do not crash....

summary: - Random Inf crash
+ WaffleGrid Crashes innodb if read-ahead enabled
Revision history for this message
MadMatt (yonkovim) wrote :

Verified bug with single read/write thread as well..

Revision history for this message
MadMatt (yonkovim) wrote :

Fixed in Cream 0.5

Changed in wafflegrid:
assignee: nobody → MadMatt (yonkovim)
importance: Undecided → High
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.