handle_fatal_signal (sig=11) in rbt_free_node at ut0rbt.c:731 | Writing larger than 4GB bitmap file fails; bitmap write error causes heap corruption
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona Server moved to https://jira.percona.com/projects/PS |
Fix Released
|
High
|
Laurynas Biveinis | ||
5.1 |
Fix Released
|
High
|
Laurynas Biveinis | ||
5.5 |
Fix Released
|
High
|
Laurynas Biveinis |
Bug Description
=======
Thread 10 (LWP 23954):
+bt
#0 0x0000000004e3969c in pthread_kill () from /lib64/
#1 0x00000000007e2779 in my_write_core (sig=11) at /ssd/QA-
#2 0x00000000006ab0ea in handle_fatal_signal (sig=11) at /ssd/QA-
#3 <signal handler called>
#4 rbt_free_node (node=0x0, nil=0x1040f170) at /ssd/QA-
#5 0x00000000009935e9 in rbt_free_node (node=0x1040f1e0, nil=0x1040f170) at /ssd/QA-
#6 0x00000000009935e9 in rbt_free_node (node=0x1017d7d0, nil=0x1040f170) at /ssd/QA-
#7 0x0000000000993693 in rbt_free (tree=0x1017d760) at /ssd/QA-
#8 0x000000000094e1ad in log_online_
#9 0x0000000000860a76 in srv_redo_
#10 0x0000000004e34851 in start_thread () from /lib64/
#11 0x0000000005bfa11d in clone () from /lib64/libc.so.6
=======
130126 13:58:08 InnoDB: Completed initialization of buffer pool
130126 13:58:08 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130126 13:58:08 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: last tracked LSN in './ib_modified_
InnoDB: continuing tracking changed pages from LSN 1597945
130126 13:58:12 InnoDB: Waiting for the background threads to start
130126 13:58:13 Percona XtraDB (http://
130126 13:58:14 [Warning] /ssd/Percona-
130126 13:58:14 [Warning] /ssd/Percona-
130126 13:58:14 [Warning] /ssd/Percona-
130126 13:58:14 [Note] Server hostname (bind-address): '0.0.0.0'; port: 13080
130126 13:58:14 [Note] - '0.0.0.0' resolves to '0.0.0.0';
130126 13:58:14 [Note] Server socket created on IP: '0.0.0.0'.
130126 13:58:14 [Warning] 'user' entry '<email address hidden>' ignored in --skip-name-resolve mode.
130126 13:58:14 [Warning] 'proxies_priv' entry '@ <email address hidden>' ignored in --skip-name-resolve mode.
130126 13:58:15 [Note] Event Scheduler: Loaded 0 events
130126 13:58:15 [Note] /ssd/Percona-
Version: '5.5.29-
InnoDB: DEBUG: update_statistics for test/table100_
InnoDB: DEBUG: update_statistics for test/table100_
InnoDB: DEBUG: update_statistics for test/table100_
130126 13:58:20 InnoDB: Error: Write to file ./ib_modified_
InnoDB: 4096 bytes should have been written, only 0 were written.
InnoDB: Operating system error number 22.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 22 means 'Invalid argument'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://
130126 13:58:20 InnoDB: Operating system error number 22 in a file operation.
InnoDB: Error number 22 means 'Invalid argument'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://
InnoDB: Error: failed writing changed page bitmap file './ib_modified_
InnoDB: Error: log tracking bitmap write failed, stopping log tracking thread!
==23942== Thread 12:
==23942== Invalid read of size 8
==23942== at 0x9935E0: rbt_free_node (ut0rbt.c:731)
==23942== by 0x9935E8: rbt_free_node (ut0rbt.c:731)
==23942== by 0x9935E8: rbt_free_node (ut0rbt.c:731)
==23942== by 0x993692: rbt_free (ut0rbt.c:746)
==23942== by 0x94E1AC: log_online_
==23942== by 0x860A75: srv_redo_
==23942== by 0x4E34850: start_thread (in /lib64/
==23942== by 0x179C76FF: ???
==23942== Address 0x8 is not stack'd, malloc'd or (recently) free'd
==23942==
10:58:20 UTC - mysqld got signal 11 ;
=======
[roel@macpro2 210222]$ cat cmd113
ps -ef | grep 'cmdrun_113' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/210222/
mkdir /ssd/210222/
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=5950 --duration=180 --querytimeout=60 --short_
=======
Note the initial crash recovery. No idea why that is happening, but I suspect it has something to do with the startup options.
Related branches
- Stewart Smith (community): Approve
- George Ormond Lorch III (community): Approve (g2)
- Laurynas Biveinis: Pending requested
-
Diff: 168 lines (+104/-15)4 files modifiedPercona-Server/mysql-test/suite/innodb_plugin/r/percona_changed_page_bmp_debug.result (+23/-0)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_page_bmp_debug-master.opt (+1/-0)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_page_bmp_debug.test (+56/-0)
Percona-Server/storage/innodb_plugin/log/log0online.c (+24/-15)
- Stewart Smith (community): Approve
- George Ormond Lorch III (community): Approve (g2)
- Laurynas Biveinis: Pending requested
-
Diff: 170 lines (+106/-15)4 files modifiedPercona-Server/mysql-test/suite/innodb/r/percona_changed_page_bmp_debug.result (+24/-0)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_page_bmp_debug-master.opt (+1/-0)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_page_bmp_debug.test (+57/-0)
Percona-Server/storage/innobase/log/log0online.c (+24/-15)
summary: |
- handle_fatal_signal (sig=11) in rbt_free_node at ut0rbt.c:731 + Writing larger than 4GB bitmap file fails; bitmap write error causes + heap corruption |
summary: |
Writing larger than 4GB bitmap file fails; bitmap write error causes - heap corruption + heap corruption | Was: handle_fatal_signal (sig=11) in rbt_free_node at + ut0rbt.c:731 |
summary: |
- Writing larger than 4GB bitmap file fails; bitmap write error causes - heap corruption | Was: handle_fatal_signal (sig=11) in rbt_free_node at - ut0rbt.c:731 + handle_fatal_signal (sig=11) in rbt_free_node at ut0rbt.c:731 | Writing + larger than 4GB bitmap file fails; bitmap write error causes heap + corruption |
tags: | added: xtradb |
The first issue here is a a spectacular typo that causes the write to fail:
130126 13:58:20 InnoDB: Error: Write to file ./ib_modified_ log_2_8204. xdb failed at offset 35184372088832 8192.
success = os_file_ write(log_ bmp_sys- >out.name, log_bmp_ sys->out. file, (log_bmp_ sys->out. offset & 0xFFFFFFFF), (log_bmp_ sys->out. offset << 32), PAGE_BLOCK_ SIZE);
block,
(ulint)
(ulint)
MODIFIED_
s/<</>> !