InnoDB: Warning: a long semaphore wait: --Thread 140488115132160 has waited at row0upd.c line 149 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x10818e0 '&dict_operation_lock' a writer (thread id 140469879518976) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0upd.c line 149 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/row/row0mysql.c line 3270 InnoDB: Warning: a long semaphore wait: --Thread 140488115664640 has waited at buf0lru.c line 1093 for 241.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140469867538176 has waited at buf0lru.c line 1093 for 241.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140469867271936 has waited at buf0buf.ic line 372 for 241.00 seconds the semaphore: Mutex at 0x42cb0a8 '&buf_pool->zip_mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140488116463360 has waited at buf0lru.c line 1093 for 241.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140469868336896 has waited at buf0lru.c line 1093 for 241.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140469868869376 has waited at buf0buf.c line 3716 for 241.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server-5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 InnoDB: Warning: a long semaphore wait: --Thread 140488121255680 has waited at buf0buf.c line 2529 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server-5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 InnoDB: Warning: a long semaphore wait: --Thread 140469784704768 has waited at buf0buf.c line 2529 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server-5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 InnoDB: Warning: a long semaphore wait: --Thread 140469906888448 has waited at buf0flu.c line 1896 for 241.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 InnoDB: Warning: a long semaphore wait: --Thread 140469896398592 has waited at buf0buf.c line 2529 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server-5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 120711 15:25:32 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 22 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 853983 1_second, 853972 sleeps, 74347 10_second, 113893 background, 113893 flush srv_master_thread log flush and writes: 1327754 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 14195358, signal count 26569090 --Thread 140469791094528 has waited at row0upd.c line 149 for 254.00 seconds the semaphore: S-lock on RW-latch at 0x10818e0 '&dict_operation_lock' a writer (thread id 140469879518976) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0upd.c line 149 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/row/row0mysql.c line 3270 --Thread 140469873129216 has waited at buf0buf.c line 4111 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469867804416 has waited at buf0lru.c line 1093 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469873927936 has waited at buf0buf.c line 2811 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469879518976 has waited at buf0lru.c line 395 for 254.00 seconds the semaphore: Mutex at 0x42cb0a8 '&buf_pool->zip_mutex', lock var 1 waiters flag 1 --Thread 140488115132160 has waited at row0upd.c line 149 for 254.00 seconds the semaphore: S-lock on RW-latch at 0x10818e0 '&dict_operation_lock' a writer (thread id 140469879518976) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0upd.c line 149 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/row/row0mysql.c line 3270 --Thread 140488115664640 has waited at buf0lru.c line 1093 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469867538176 has waited at buf0lru.c line 1093 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469867271936 has waited at buf0buf.ic line 372 for 254.00 seconds the semaphore: Mutex at 0x42cb0a8 '&buf_pool->zip_mutex', lock var 1 waiters flag 1 --Thread 140488116463360 has waited at buf0lru.c line 1093 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469868336896 has waited at buf0lru.c line 1093 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469868869376 has waited at buf0buf.c line 3716 for 254.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140488121255680 has waited at buf0buf.c line 2529 for 254.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140469784704768 has waited at buf0buf.c line 2529 for 254.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140469906888448 has waited at buf0flu.c line 1896 for 254.00 seconds the semaphore: Mutex at 0x42cb068 '&buf_pool->mutex', lock var 1 waiters flag 1 --Thread 140469896398592 has waited at buf0buf.c line 2529 for 254.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140488126048000 has waited at buf0buf.c line 2529 for 252.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140488125249280 has waited at buf0buf.c line 2529 for 243.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140469864077056 has waited at buf0buf.c line 2529 for 183.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140469864077056 has waited at buf0buf.c line 2529 for 183.00 seconds the semaphore: S-lock on RW-latch at 0x42cb128 '&buf_pool->page_hash_latch' a writer (thread id 140469868869376) has reserved it in mode wait exclusive number of readers 1, waiters flag 1, lock_word: ffffffffffffffff Last time read locked in file buf0buf.c line 2529 Last time write locked in file /home/jenkins/workspace/percona-server-5.5-rpms/label_exp/centos6-64/target/BUILD/Percona-Server-5.5.24-rel26.0/Percona-Server -5.5.24-rel26.0/storage/innobase/buf/buf0buf.c line 3479 --Thread 140469862745856 has waited at dict0dict.c line 744 for 183.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469948847872 has waited at srv0srv.c line 2678 for 183.00 seconds the semaphore: Mutex at 0x2dce368 '&kernel_mutex', lock var 1 waiters flag 1 --Thread 140469863544576 has waited at dict0dict.c line 744 for 183.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140488116197120 has waited at dict0dict.c line 744 for 179.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469877389056 has waited at dict0dict.c line 744 for 179.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469864875776 has waited at dict0dict.c line 744 for 179.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140488121788160 has waited at dict0dict.c line 744 for 178.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140488122320640 has waited at dict0dict.c line 744 for 178.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469785769728 has waited at dict0dict.c line 744 for 178.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469882713856 has waited at dict0dict.c line 744 for 174.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469862213376 has waited at dict0dict.c line 744 for 174.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469871798016 has waited at dict0dict.c line 744 for 174.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469787100928 has waited at dict0dict.c line 744 for 174.00 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469790562048 has waited at trx0trx.c line 229 for 151.00 seconds the semaphore: Mutex at 0x2dce368 '&kernel_mutex', lock var 1 waiters flag 1 --Thread 140488122853120 has waited at dict0dict.c line 744 for 91.000 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 --Thread 140469881116416 has waited at trx0trx.c line 229 for 73.000 seconds the semaphore: Mutex at 0x2dce368 '&kernel_mutex', lock var 1 waiters flag 1 --Thread 140469785237248 has waited at dict0dict.c line 744 for 35.000 seconds the semaphore: Mutex at 0x1f2d23a8 '&dict_sys->mutex', lock var 1 waiters flag 1 Mutex spin waits 245427319, rounds 319745618, OS waits 1338081 RW-shared spins 18744327, rounds 163905846, OS waits 3904949 RW-excl spins 10071607, rounds 352834025, OS waits 8551813 Spin rounds per wait: 1.30 mutex, 8.74 RW-shared, 35.03 RW-excl -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 1215107 OS file reads, 8143869 OS file writes, 189715 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 327, seg size 329, 44593 merges merged operations: insert 1036, delete mark 448009, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 35401589, node heap has 25498 buffer(s) 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 481571207335 Log flushed up to 481571207335 Last checkpoint at 480641341774 Max checkpoint age 1303885210 Checkpoint age target 1263138798 Modified age 922286129 Checkpoint age 929865561 0 pending log writes, 0 pending chkp writes 5265596 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 17649631232; in additional pool allocated 0 InnoDB: ###### Diagnostic info printed to the standard error stream