Go on with... I tested MySQL 5.1.37 next. Its surprise that no crash happened when I ran 10 warehouses(MySQL 5.0.75 crashed). Then I found that when running 10 warehouses on MySQL 5.1.37, the TPM is only ~20k, but ~40k on MySQL 5.0.75(I dont know what caused this drop, SSD utility is very low). So I increased the warehouse number to 100, and the TPM go up to 40k soon, when I do hotbackup. As a result, the MySQL 5.1.7 crashed !!! To repeat the bug , you have to promise that your MySQL has high TPM(a least ~40k) when running DBT2 test, no matter how many warehouses you use. Version: '5.1.37' socket: '/schooner/data/db0/mysql.sock' port: 3307 Source distribution InnoDB: Error (2): trying to extend a single-table tablespace 3 InnoDB: by single page(s) though the space size 203648. Page no 203648. 091027 14:38:30InnoDB: Error: trying to access a stray pointer (nil) InnoDB: buf pool start is at 0x7fc0a2b4c000, end at 0x7fc4a2b4c000 InnoDB: Probable reason is database corruption or memory InnoDB: corruption. If this happens in an InnoDB database recovery, see InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: how to force recovery. 091027 14:38:30 InnoDB: Assertion failure in thread 1086601536 in file ../../storage/innobase/include/buf0buf.ic line 225 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html InnoDB: about forcing recovery. 091027 14:38:30 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=134217728 read_buffer_size=1048576 max_used_connections=33 max_threads=1000 threads_connected=32 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3213236 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0x2056fa60 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x40c430f0 thread_stack 0x40000 /usr/local/mysql-5.1-product//libexec/mysqld(my_print_stacktrace+0x32)[0x873db0] /usr/local/mysql-5.1-product//libexec/mysqld(handle_segfault+0x322)[0x5a2472] /lib64/libpthread.so.0[0x331400de80] /lib64/libc.so.6(gsignal+0x35)[0x3313430155] /lib64/libc.so.6(abort+0x110)[0x3313431bf0] /usr/local/mysql-5.1-product//libexec/mysqld[0x78aec8] /usr/local/mysql-5.1-product//libexec/mysqld[0x78c269] /usr/local/mysql-5.1-product//libexec/mysqld(page_create+0x58)[0x78bdde] /usr/local/mysql-5.1-product//libexec/mysqld[0x7db2c9] /usr/local/mysql-5.1-product//libexec/mysqld(btr_page_split_and_insert+0x19f)[0x7de24c] /usr/local/mysql-5.1-product//libexec/mysqld(btr_cur_pessimistic_insert+0x20e)[0x72f5ad] /usr/local/mysql-5.1-product//libexec/mysqld(row_ins_index_entry_low+0x407)[0x79bfa6] /usr/local/mysql-5.1-product//libexec/mysqld(row_ins_index_entry+0xc7)[0x79c30e] /usr/local/mysql-5.1-product//libexec/mysqld[0x79c4d0] /usr/local/mysql-5.1-product//libexec/mysqld(row_ins+0xa0)[0x79c57a] /usr/local/mysql-5.1-product//libexec/mysqld(row_ins_step+0x18f)[0x79cb7a] /usr/local/mysql-5.1-product//libexec/mysqld(row_insert_for_mysql+0x225)[0x79eb24] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN11ha_innobase9write_rowEPh+0xff)[0x727c6f] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN7handler12ha_write_rowEPh+0x6f)[0x695c4f] /usr/local/mysql-5.1-product//libexec/mysqld(_Z12write_recordP3THDP8st_tableP12st_copy_info+0x5d)[0x62846d] /usr/local/mysql-5.1-product//libexec/mysqld(_Z12mysql_insertP3THDP10TABLE_LISTR4ListI4ItemERS3_IS5_ES6_S6_15enum_duplicatesb+0x9fa)[0x62d51a] /usr/local/mysql-5.1-product//libexec/mysqld(_Z21mysql_execute_commandP3THD+0x1eed)[0x5b224d] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x6eeafc] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0xf7)[0x6f1567] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x5d6)[0x6f39a6] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN7sp_head7executeEP3THD+0x441)[0x6f23d1] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x4a7)[0x6f4e87] /usr/local/mysql-5.1-product//libexec/mysqld(_Z21mysql_execute_commandP3THD+0x11b1)[0x5b1511] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x6eeafc] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0xf7)[0x6f1567] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x5d6)[0x6f39a6] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN7sp_head7executeEP3THD+0x441)[0x6f23d1] /usr/local/mysql-5.1-product//libexec/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x4a7)[0x6f4e87] /usr/local/mysql-5.1-product//libexec/mysqld(_Z21mysql_execute_commandP3THD+0x11b1)[0x5b1511] /usr/local/mysql-5.1-product//libexec/mysqld(_Z11mysql_parseP3THDPKcjPS2_+0x228)[0x5b8128] /usr/local/mysql-5.1-product//libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x3e3)[0x5b8513] /usr/local/mysql-5.1-product//libexec/mysqld(_Z10do_commandP3THD+0xe4)[0x5b96c4] /usr/local/mysql-5.1-product//libexec/mysqld(handle_one_connection+0x5b7)[0x5aa077] /lib64/libpthread.so.0[0x3314006307] /lib64/libc.so.6(clone+0x6d)[0x33134d1ded] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x7fc058282ec8 is an invalid pointer thd->thread_id=10 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file /usr/local/mysql-5.1-product/bin/mysqld_safe: line 137: 23584 Aborted (core dumped) nohup /usr/local/mysql-5.1-product//libexec/mysqld --defaults-file=/opt/schooner/mysql/config/my.cnf --basedir=/usr/local/mysql-5.1 --datadir=/schooner/data/db0 --user=root --log-error=/schooner/data/db0/localhost.localdomain.err --pid-file=/schooner/data/db0/localhost.localdomain.pid --socket=/schooner/data/db0/mysql.sock --port=3307 < /dev/null >> /schooner/data/db0/localhost.localdomain.err 2>&1 091027 14:38:35 mysqld_safe Number of processes running now: 0 091027 14:38:35 mysqld_safe mysqld restarted 091027 14:38:35 mysqld_safe Number of processes running now: 0 091027 14:38:35 mysqld_safe mysqld restarted InnoDB: Log scan progressed past the checkpoint lsn 2 4258902181 091027 14:38:41 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 2 4261786095 InnoDB: 18 transaction(s) which must be rolled back or cleaned up InnoDB: in total 346 row operations to undo InnoDB: Trx id counter is 0 718336 091027 14:38:43 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Starting in background the rollback of uncommitted transactions 091027 14:39:23 InnoDB: Rolling back trx with id 0 717860, 1 rows to undo 091027 14:39:23 InnoDB: Started; log sequence number 2 4261786095 InnoDB: Rolling back of trx id 0 717860 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717857, 3 rows to undo InnoDB: Rolling back of trx id 0 717857 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717855, 3 rows to undo 091027 14:39:23 [Note] Event Scheduler: Loaded 0 events 091027 14:39:23 [Note] /usr/local/mysql-5.1-product//libexec/mysqld: ready for connections. Version: '5.1.37' socket: '/schooner/data/db0/mysql.sock' port: 3307 Source distribution InnoDB: Rolling back of trx id 0 717855 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717854, 11 rows to undo InnoDB: Rolling back of trx id 0 717854 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717853, 3 rows to undo InnoDB: Rolling back of trx id 0 717853 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717850, 7 rows to undo InnoDB: Rolling back of trx id 0 717850 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717849, 15 rows to undo InnoDB: Rolling back of trx id 0 717849 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717846, 13 rows to undo InnoDB: Rolling back of trx id 0 717846 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717845, 14 rows to undo InnoDB: Rolling back of trx id 0 717845 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717844, 9 rows to undo InnoDB: Rolling back of trx id 0 717844 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717841, 9 rows to undo InnoDB: Rolling back of trx id 0 717841 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717839, 5 rows to undo InnoDB: Rolling back of trx id 0 717839 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717836, 25 rows to undo InnoDB: Rolling back of trx id 0 717836 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717835, 13 rows to undo InnoDB: Rolling back of trx id 0 717835 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717821, 19 rows to undo InnoDB: Rolling back of trx id 0 717821 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717811, 15 rows to undo InnoDB: Rolling back of trx id 0 717811 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717744, 65 rows to undo InnoDB: Rolling back of trx id 0 717744 completed 091027 14:39:23 InnoDB: Rolling back trx with id 0 717728, 116 rows to undo InnoDB: Rolling back of trx id 0 717728 completed 091027 14:39:23 InnoDB: Rollback of non-prepared transactions completed