InnoDB: Failing assertion: node->n_pending == 0 in file fil0fil.c line 2415 | abort in fil_delete_tablespace
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona Server moved to https://jira.percona.com/projects/PS |
Invalid
|
High
|
Laurynas Biveinis | ||
5.1 |
Invalid
|
Undecided
|
Laurynas Biveinis | ||
5.5 |
Fix Released
|
High
|
Laurynas Biveinis | ||
5.6 |
Invalid
|
Undecided
|
Laurynas Biveinis | ||
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC |
Fix Released
|
High
|
Unassigned |
Bug Description
We encountered a crash of percona server under high load. This is after a week or 2 since we updated percona. Versions used:
Percona-
Percona-
percona-
Percona-
percona-
Percona-
Percona-
The error log captured this:
130214 16:08:15 [Warning] IP address '10.208.39.190' could not be resolved: Name or service not known
130218 2:23:36 InnoDB: Assertion failure in thread 140442299004672 in file fil0fil.c line 2415
InnoDB: Failing assertion: node->n_pending == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://
InnoDB: about forcing recovery.
01:23:36 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://
key_buffer_
read_buffer_
max_used_
max_threads=1500
thread_count=118
connection_
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x12d29020
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fbb4557ee58 thread_stack 0x40000
/usr/sbin/
/usr/sbin/
/lib64/
/lib64/
/lib64/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/usr/sbin/
/lib64/
/lib64/
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fb8145aa830): is an invalid pointer
Connection ID (thread ID): 3239136
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://
in the manual which will help you identify the cause of the crash.
130218 02:23:37 mysqld_safe Number of processes running now: 0
130218 02:23:37 mysqld_safe mysqld restarted
130218 2:23:37 [Warning] The syntax '--log-
130218 2:23:37 [Note] Plugin 'FEDERATED' is disabled.
130218 2:23:37 InnoDB: The InnoDB memory heap is disabled
130218 2:23:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130218 2:23:37 InnoDB: Compressed tables use zlib 1.2.3
130218 2:23:37 InnoDB: Using Linux native AIO
130218 2:23:37 InnoDB: Initializing buffer pool, size = 9.0G
InnoDB: HugeTLB: Warning: Failed to allocate 9927917568 bytes. errno 1
InnoDB HugeTLB: Warning: Using conventional memory pool
130218 2:23:38 InnoDB: Completed initialization of buffer pool
130218 2:23:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 12042613609937
130218 2:23:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 12042618852352
InnoDB: Doing recovery: scanned up to log sequence number 12042624095232
InnoDB: Doing recovery: scanned up to log sequence number 12042629338112
InnoDB: Doing recovery: scanned up to log sequence number 12042634580992
InnoDB: Doing recovery: scanned up to log sequence number 12042639823872
InnoDB: Doing recovery: scanned up to log sequence number 12042645066752
InnoDB: Doing recovery: scanned up to log sequence number 12042650309632
InnoDB: Doing recovery: scanned up to log sequence number 12042655552512
InnoDB: Doing recovery: scanned up to log sequence number 12042660795392
InnoDB: Doing recovery: scanned up to log sequence number 12042666038272
InnoDB: Doing recovery: scanned up to log sequence number 12042671281152
InnoDB: Doing recovery: scanned up to log sequence number 12042676524032
InnoDB: Doing recovery: scanned up to log sequence number 12042681766912
InnoDB: Doing recovery: scanned up to log sequence number 12042687009792
InnoDB: Doing recovery: scanned up to log sequence number 12042692252672
InnoDB: Doing recovery: scanned up to log sequence number 12042697495552
InnoDB: Doing recovery: scanned up to log sequence number 12042702738432
InnoDB: Doing recovery: scanned up to log sequence number 12042707981312
InnoDB: Doing recovery: scanned up to log sequence number 12042713224192
InnoDB: Doing recovery: scanned up to log sequence number 12042718467072
InnoDB: Doing recovery: scanned up to log sequence number 12042723709952
InnoDB: Doing recovery: scanned up to log sequence number 12042728952832
InnoDB: Doing recovery: scanned up to log sequence number 12042734195712
InnoDB: Doing recovery: scanned up to log sequence number 12042739438592
InnoDB: Doing recovery: scanned up to log sequence number 12042744681472
InnoDB: Doing recovery: scanned up to log sequence number 12042749924352
InnoDB: Doing recovery: scanned up to log sequence number 12042755167232
InnoDB: Doing recovery: scanned up to log sequence number 12042760410112
InnoDB: Doing recovery: scanned up to log sequence number 12042765652992
InnoDB: Doing recovery: scanned up to log sequence number 12042770895872
InnoDB: Doing recovery: scanned up to log sequence number 12042776138752
InnoDB: Doing recovery: scanned up to log sequence number 12042781381632
InnoDB: Doing recovery: scanned up to log sequence number 12042786624512
InnoDB: Doing recovery: scanned up to log sequence number 12042791867392
InnoDB: Doing recovery: scanned up to log sequence number 12042797110272
InnoDB: Doing recovery: scanned up to log sequence number 12042802353152
InnoDB: Doing recovery: scanned up to log sequence number 12042807596032
InnoDB: Doing recovery: scanned up to log sequence number 12042812838912
InnoDB: Doing recovery: scanned up to log sequence number 12042818081792
InnoDB: Doing recovery: scanned up to log sequence number 12042823324672
InnoDB: Doing recovery: scanned up to log sequence number 12042828567552
InnoDB: Doing recovery: scanned up to log sequence number 12042833810432
InnoDB: Doing recovery: scanned up to log sequence number 12042839053312
InnoDB: Doing recovery: scanned up to log sequence number 12042844296192
InnoDB: Doing recovery: scanned up to log sequence number 12042849539072
InnoDB: Doing recovery: scanned up to log sequence number 12042854781952
InnoDB: Doing recovery: scanned up to log sequence number 12042860024832
InnoDB: Doing recovery: scanned up to log sequence number 12042865267712
InnoDB: Doing recovery: scanned up to log sequence number 12042870510592
InnoDB: Doing recovery: scanned up to log sequence number 12042875753472
InnoDB: Doing recovery: scanned up to log sequence number 12042880996352
InnoDB: Doing recovery: scanned up to log sequence number 12042886239232
InnoDB: Doing recovery: scanned up to log sequence number 12042891482112
InnoDB: Doing recovery: scanned up to log sequence number 12042896724992
InnoDB: Doing recovery: scanned up to log sequence number 12042901967872
InnoDB: Doing recovery: scanned up to log sequence number 12042907210752
InnoDB: Doing recovery: scanned up to log sequence number 12042912453632
InnoDB: Doing recovery: scanned up to log sequence number 12042917696512
InnoDB: Doing recovery: scanned up to log sequence number 12042922939392
InnoDB: Doing recovery: scanned up to log sequence number 12042928182272
InnoDB: Doing recovery: scanned up to log sequence number 12042933425152
InnoDB: Doing recovery: scanned up to log sequence number 12042938668032
InnoDB: Doing recovery: scanned up to log sequence number 12042943910912
InnoDB: Doing recovery: scanned up to log sequence number 12042949153792
InnoDB: Doing recovery: scanned up to log sequence number 12042954396672
InnoDB: Doing recovery: scanned up to log sequence number 12042959639552
InnoDB: Doing recovery: scanned up to log sequence number 12042964882432
InnoDB: Doing recovery: scanned up to log sequence number 12042970125312
InnoDB: Doing recovery: scanned up to log sequence number 12042975368192
InnoDB: Doing recovery: scanned up to log sequence number 12042980611072
InnoDB: Doing recovery: scanned up to log sequence number 12042985853952
InnoDB: Doing recovery: scanned up to log sequence number 12042991096832
InnoDB: Doing recovery: scanned up to log sequence number 12042996339712
InnoDB: Doing recovery: scanned up to log sequence number 12043001582592
InnoDB: Doing recovery: scanned up to log sequence number 12043006825472
InnoDB: Doing recovery: scanned up to log sequence number 12043012068352
InnoDB: Doing recovery: scanned up to log sequence number 12043017311232
InnoDB: Doing recovery: scanned up to log sequence number 12043022554112
InnoDB: Doing recovery: scanned up to log sequence number 12043027796992
InnoDB: Doing recovery: scanned up to log sequence number 12043033039872
InnoDB: Doing recovery: scanned up to log sequence number 12043038282752
InnoDB: Doing recovery: scanned up to log sequence number 12043043525632
InnoDB: Doing recovery: scanned up to log sequence number 12043048768512
InnoDB: Doing recovery: scanned up to log sequence number 12043054011392
InnoDB: Doing recovery: scanned up to log sequence number 12043059254272
InnoDB: Doing recovery: scanned up to log sequence number 12043064497152
InnoDB: Doing recovery: scanned up to log sequence number 12043069740032
InnoDB: Doing recovery: scanned up to log sequence number 12043074982912
InnoDB: Doing recovery: scanned up to log sequence number 12043080225792
InnoDB: Doing recovery: scanned up to log sequence number 12043085468672
InnoDB: Doing recovery: scanned up to log sequence number 12043090711552
InnoDB: Doing recovery: scanned up to log sequence number 12043095954432
InnoDB: Doing recovery: scanned up to log sequence number 12043101197312
InnoDB: Doing recovery: scanned up to log sequence number 12043106440192
InnoDB: Doing recovery: scanned up to log sequence number 12043111683072
InnoDB: Doing recovery: scanned up to log sequence number 12043114035027
InnoDB: in total 11 row operations to undo
InnoDB: Trx id counter is A01D1A00
130218 2:24:02 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
130218 2:24:04 InnoDB: Rolling back trx with id A01D18EE, 11 rows to undo
InnoDB: Dropping table with id 39791 in recovery if it exists
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Rolling back of trx id A01D18EE completed
130218 2:24:04 InnoDB: Waiting for the background threads to start
130218 2:24:05 Percona XtraDB (http://
130218 2:24:05 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
130218 2:24:05 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130218 2:24:05 [Note] Server socket created on IP: '0.0.0.0'.
130218 2:24:05 [Note] Event Scheduler: Loaded 0 events
130218 2:24:05 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.29-29.4-log' socket: '/tmp/mysql.sock' port: 3306 Percona Server (GPL), Release rel29.4, Revision 401
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
InnoDB: Error: trying to load index PRIMARY for table lcg_frFR/
InnoDB: but the index tree has been freed!
<snip. this goes on for a long while>
Please let us know if we can provide any other information to assist in finding the cause for this. If it happens again we'll update this ticket.
Related branches
- Alexey Kopytov (community): Approve
-
Diff: 406 lines (+14/-185) (has conflicts)10 files modifiedPercona-Server/storage/innobase/buf/buf0buf.c (+0/-74)
Percona-Server/storage/innobase/buf/buf0flu.c (+1/-8)
Percona-Server/storage/innobase/buf/buf0lru.c (+0/-40)
Percona-Server/storage/innobase/fil/fil0fil.c (+0/-49)
Percona-Server/storage/innobase/handler/ha_innodb.cc (+8/-2)
Percona-Server/storage/innobase/include/buf0buf.h (+5/-0)
Percona-Server/storage/innobase/include/buf0buf.ic (+0/-1)
Percona-Server/storage/innobase/include/buf0lru.h (+0/-7)
Percona-Server/storage/innobase/include/srv0srv.h (+0/-2)
Percona-Server/storage/innobase/srv/srv0srv.c (+0/-2)
Changed in percona-xtradb-cluster: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in percona-xtradb-cluster: | |
status: | Triaged → Fix Released |
Btw, dmesg and kernel message log did not show any events related to hardware problems. Hardware health seems fine.