I want to track all the related data we are collecting and all the things we try in this one bug.
Basically under heavy load, innodb will start to throw random errors in secondary indexes. Here is an example of the error with some extra logging turned on:
Yves has managed to get this to fail with 1 single warehouse on DBT2. In order to do this you need to use a tmpfs ( fast disk ) and store everything on it.
I have gotten this to appear after about 20 minutes of running a 20W test with regular disk and a 768M BP....
We appear to be hitting some event. Possibly a flush or some process that only wakes up after so many changes.
Additionally we think we may have missed a call the invalidates, reads a page, moves a block/page, or does something outside of buf_LRU_search_and_free_block & buf_read_page_low ....
I want to track all the related data we are collecting and all the things we try in this one bug.
Basically under heavy load, innodb will start to throw random errors in secondary indexes. Here is an example of the error with some extra logging turned on:
49332298- 2994417918- Innodb inside btr_cur_ search_ to_nth_ level index myidx1 thread: 1160882512 2994417994- ... S1: space: 5 page_no 4 : thread : 1160882512 2994418044- ... S1: space: 3 page_no 40949 : thread : 1100904784 2994418098- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1080850768 2994418175- ... S1: space: 5 page_no 3 : thread : 1080850768 2994418225- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100904784 2994418302- ... S1: space: 3 page_no 3 : thread : 1100904784 2994418352- ... S1: space: 5 page_no 1062 : thread : 1080850768 2994418405- ... S1: space: 3 page_no 54 : thread : 1100904784 2994418456- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100106064 2994418533- ... S1: space: 3 page_no 40949 : thread : 1100904784 2994418587- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100904784 2994418664- ... S1: space: 3 page_no 3 : thread : 1100904784 2994418714- ... S1: space: 3 page_no 54 : thread : 1100904784 2994418765- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1077369168 2994418842- ... S1: space: 3 page_no 40949 : thread : 1100904784 2994418896- ... S1: space: 7 page_no 3 : thread : 1077369168 2994418946- ... S1: space: 7 page_no 329 : thread : 1077369168 2994418998- ... S1: space: 7 page_no 2825 : thread : 1077369168 2994419051- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100106064 2994419128: InnoDB: error in sec index entry update in 2994419171- InnoDB: index `myidx1` of table `dbt2`... S1: space: 3 page_no 57834 : thread : 1080318288 2994419263- Innodb inside btr_cur_ search_ to_nth_ level index GEN_CLUST_INDEX thread: 1100372304 2994419348- ... S1: space: 6 page_no 3 : thread : 1100372304 2994419398- ... S1: space: 6 page_no 41 : thread : 1100372304 2994419449- ... S1: space: 6 page_no 5974 : thread : 1100372304 2994419502- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100106064 2994419579- ... S1: space: 7 page_no 3 : thread : 1100106064 2994419629- .`new_order` 2994419642- InnoDB: tuple DATA TUPLE: 3 fields; 2994419678- 0: len 4; hex 80000006; asc ... S1: space: 7 page_no 358 : thread : 1100106064 2994419763- ... S1: space: 7 page_no 17866 : thread : 1100106064 2994419817- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100106064 2994419894- ... S1: space: 7 page_no 3 : thread : 1100106064 2994419944- ;; 1: len 4; hex 80000efd; asc ... S1: space: 1 page_no 57 : thread : 1081928016 2994420029- ... S1: space: 1 page_no 17623 : thread : 1081928016 2994420083- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100638544 2994420160- ... S1: space: 5 page_no 3 : thread : 1100638544 2994420210- ... S1: space: 5 page_no 849 : thread : 1100638544 2994420262- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1080318288 2994420339- Innodb inside btr_cur_ search_ to_nth_ level index PRIMARY thread: 1100638544
49332299-
49332300-
49332301-
49332302-
49332303-
49332304-
49332305-
49332306-
49332307-
49332308-
49332309-
49332310-
49332311-
49332312-
49332313-
49332314-
49332315-
49332316-
49332317-
49332318:
49332319-
49332320-
49332321-
49332322-
49332323-
49332324-
49332325-
49332326-
49332327-
49332328-
49332329-
49332330-
49332331-
49332332-
49332333-
49332334-
49332335-
49332336-
49332337-
49332338-
This will eventually crash the machine.
Yves has managed to get this to fail with 1 single warehouse on DBT2. In order to do this you need to use a tmpfs ( fast disk ) and store everything on it.
I have gotten this to appear after about 20 minutes of running a 20W test with regular disk and a 768M BP....
We appear to be hitting some event. Possibly a flush or some process that only wakes up after so many changes.
Additionally we think we may have missed a call the invalidates, reads a page, moves a block/page, or does something outside of buf_LRU_ search_ and_free_ block & buf_read_page_low ....