complete_io does seem to fix some issues... however it raises others:
root@frankenmatt:/benchmarks/waffle_2.0# grep -n "7:1 " /mysql_home/mysql540-waffle/error.log 51:Memcached miss: Block: bf2:7:1 : Thread : 1144666448 time:51 52:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:468 53:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:8401 147431:Memcached get: Block: bf2:7:1 : Thread : 1144666448 time:125 147434:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:6028 185727:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:82 454229:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:81 743017:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:137 1559529:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:77 2209311:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:85 root@frankenmatt:/benchmarks/waffle_2.0# grep -n "7:3 " /mysql_home/mysql540-waffle/error.log 50:Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:382 147430:Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:52 root@frankenmatt:/benchmarks/waffle_2.0# grep -i -n -b3 error /mysql_home/mysql540-waffle/error.log 147429-9567137-Memcached miss: Block: bf2:0:2 : Thread : 1144666448 time:0 147430-9567198-Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:52 147431-9567260-Memcached get: Block: bf2:7:1 : Thread : 1144666448 time:125 147432:9567322:090528 11:45:49 InnoDB: Error: space id and page n:o stored in the page 147433-9567395-InnoDB: read in are 7:3, should be 7:1! 147434-9567435-Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:6028 147435-9567498-Memcached miss: Block: bf2:7:147 : Thread : 1144666448 time:43 -- 147477-9570174-Memcached miss: Block: bf2:1:27 : Thread : 1144932688 time:51 147478-9570237-Memcached set: Block: bf2:2:116 : Thread : 1145198928 time:15078 147479-9570303-Memcached get: Block: bf2:1:1 : Thread : 1144666448 time:126 147480:9570365:090528 11:45:49 InnoDB: Error: space id and page n:o stored in the page 147481-9570438-InnoDB: read in are 1:3, should be 1:1! 147482-9570478-Memcached set: Block: bf2:1:1 : Thread : 1144666448 time:16129 147483-9570542-Memcached miss: Block: bf2:3:3 : Thread : 1144666448 time:53 -- 149045-9670798-Memcached get: Block: bf2:1:e8d : Thread : 1147861328 time:153 149046-9670862-Memcached set: Block: bf2:3:b8f : Thread : 1147062608 time:24442 149047-9670928-Memcached get: Block: bf2:7:4001 : Thread : 1146263888 time:223 149048:9670993:090528 11:45:51 InnoDB: Error: space id and page n:o stored in the page 149049-9671066-InnoDB: read in are 7:22621, should be 7:16385! 149050-9671114-Memcached set: Block: bf2:7:4001 : Thread : 1146263888 time:3318 149051-9671180-Memcached miss: Block: bf2:7:432c : Thread : 1146263888 time:48 -- 174720-11338188-Memcached set: Block: bf2:1:63ca : Thread : 1145731408 time:8270 174721-11338254-Memcached get: Block: bf2:1:85a2 : Thread : 1145731408 time:161 174722-11338319-Memcached get: Block: bf2:1:8001 : Thread : 1145731408 time:121 174723:11338384:090528 11:46:11 InnoDB: Error: space id and page n:o stored in the page 174724-11338457-InnoDB: read in are 1:37219, should be 1:32769! 174725-11338505-Memcached get: Block: bf2:1:7aa0 : Thread : 1144932688 time:279 174726-11338570-Memcached miss: Block: bf2:3:5dc0 : Thread : 1145731408 time:74
complete_io does seem to fix some issues... however it raises others:
root@frankenmat t:/benchmarks/ waffle_ 2.0# grep -n "7:1 " /mysql_ home/mysql540- waffle/ error.log t:/benchmarks/ waffle_ 2.0# grep -n "7:3 " /mysql_ home/mysql540- waffle/ error.log t:/benchmarks/ waffle_ 2.0# grep -i -n -b3 error /mysql_ home/mysql540- waffle/ error.log 9567137- Memcached miss: Block: bf2:0:2 : Thread : 1144666448 time:0 9567198- Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:52 9567260- Memcached get: Block: bf2:7:1 : Thread : 1144666448 time:125 9567322: 090528 11:45:49 InnoDB: Error: space id and page n:o stored in the page 9567395- InnoDB: read in are 7:3, should be 7:1! 9567435- Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:6028 9567498- Memcached miss: Block: bf2:7:147 : Thread : 1144666448 time:43 9570174- Memcached miss: Block: bf2:1:27 : Thread : 1144932688 time:51 9570237- Memcached set: Block: bf2:2:116 : Thread : 1145198928 time:15078 9570303- Memcached get: Block: bf2:1:1 : Thread : 1144666448 time:126 9570365: 090528 11:45:49 InnoDB: Error: space id and page n:o stored in the page 9570438- InnoDB: read in are 1:3, should be 1:1! 9570478- Memcached set: Block: bf2:1:1 : Thread : 1144666448 time:16129 9570542- Memcached miss: Block: bf2:3:3 : Thread : 1144666448 time:53 9670798- Memcached get: Block: bf2:1:e8d : Thread : 1147861328 time:153 9670862- Memcached set: Block: bf2:3:b8f : Thread : 1147062608 time:24442 9670928- Memcached get: Block: bf2:7:4001 : Thread : 1146263888 time:223 9670993: 090528 11:45:51 InnoDB: Error: space id and page n:o stored in the page 9671066- InnoDB: read in are 7:22621, should be 7:16385! 9671114- Memcached set: Block: bf2:7:4001 : Thread : 1146263888 time:3318 9671180- Memcached miss: Block: bf2:7:432c : Thread : 1146263888 time:48 11338188- Memcached set: Block: bf2:1:63ca : Thread : 1145731408 time:8270 11338254- Memcached get: Block: bf2:1:85a2 : Thread : 1145731408 time:161 11338319- Memcached get: Block: bf2:1:8001 : Thread : 1145731408 time:121 11338384: 090528 11:46:11 InnoDB: Error: space id and page n:o stored in the page 11338457- InnoDB: read in are 1:37219, should be 1:32769! 11338505- Memcached get: Block: bf2:1:7aa0 : Thread : 1144932688 time:279 11338570- Memcached miss: Block: bf2:3:5dc0 : Thread : 1145731408 time:74
51:Memcached miss: Block: bf2:7:1 : Thread : 1144666448 time:51
52:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:468
53:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:8401
147431:Memcached get: Block: bf2:7:1 : Thread : 1144666448 time:125
147434:Memcached set: Block: bf2:7:1 : Thread : 1144666448 time:6028
185727:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:82
454229:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:81
743017:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:137
1559529:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:77
2209311:Memcached (write) set: Block: bf2:7:1 : Thread : 1144400208 time:85
root@frankenmat
50:Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:382
147430:Memcached miss: Block: bf2:7:3 : Thread : 1144666448 time:52
root@frankenmat
147429-
147430-
147431-
147432:
147433-
147434-
147435-
--
147477-
147478-
147479-
147480:
147481-
147482-
147483-
--
149045-
149046-
149047-
149048:
149049-
149050-
149051-
--
174720-
174721-
174722-
174723:
174724-
174725-
174726-