2017-12-29T17:53:31.538067+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140290300778240 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538220+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140299701008128 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538257+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288321341184 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538290+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288658458368 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538322+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288359913216 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538354+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140299689912064 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538408+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288431773440 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538437+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288485140224 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538465+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288278542080 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538492+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140299324012288 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538519+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288462948096 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538546+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288460834560 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538573+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288343004928 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538606+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288621471488 has waited at row0ins.cc line 2540 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 2017-12-29T17:53:31.538634+08:00 0 [Warning] InnoDB: A long semaphore wait: --Thread 140288644192000 has waited at row0sel.cc line 3519 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2017-12-29 17:53:35 0x7f99cc9ef700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 1108974 srv_active, 0 srv_shutdown, 23841 srv_idle srv_master_thread log flush and writes: 1132815 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 22949078 --Thread 140299326654208 has waited at row0sel.cc line 3519 for 243.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140299298121472 has waited at row0ins.cc line 2540 for 235.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140290300778240 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288419092224 has waited at row0ins.cc line 2540 for 244.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140299701008128 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288321341184 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288658458368 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288359913216 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140299689912064 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140298667824896 has waited at buf0flu.cc line 1224 for 128.00 seconds the semaphore: SX-lock on RW-latch at 0x7f9d9ceca8b0 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode SX number of readers 0, waiters flag 1, lock_word: 10000000 Last time read locked in file not yet reserved line 0 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/fsp/fsp0fsp.cc line 167 --Thread 140288431773440 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288485140224 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288278542080 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140299684099840 has waited at row0sel.cc line 3519 for 244.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140299324012288 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288462948096 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288460834560 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288343004928 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288621471488 has waited at row0ins.cc line 2540 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 --Thread 140288644192000 has waited at row0sel.cc line 3519 for 245.00 seconds the semaphore: S-lock on RW-latch at 0x7f9d9cec2810 created in file buf0buf.cc line 1456 a writer (thread id 140299334051584) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file row0sel.cc line 3519 Last time write locked in file /mnt/workspace/percona-xtradb-cluster-5.7-redhat-binary/label/centos6-64/rpmbuild/BUILD/Percona-XtraDB-Cluster-5.7.18-29.20/storage/innobase/include/mtr0mtr.ic line 153 OS WAIT ARRAY INFO: signal count 26669276 RW-shared spins 0, rounds 6199474, OS waits 284832 RW-excl spins 0, rounds 9764616, OS waits 452584 RW-sx spins 198720, rounds 3271476, OS waits 24794 Spin rounds per wait: 6199474.00 RW-shared, 9764616.00 RW-excl, 16.46 RW-sx ------------ TRANSACTIONS ------------ Trx id counter 653883500 Purge done for trx's n:o < 653872659 undo n:o < 0 state: running but idle History list length 4336 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 421832307359328, not started 0 lock struct(s), heap size 1136, 0 row lock(s) ---TRANSACTION 421832307358120, not started ...... ---TRANSACTION 653873314, ACTIVE 235 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 137996, OS thread handle 140299298121472, query id 2106178425 10.103.22.179 developer update insert into pay_record( orderid,payamount,paystatus,paychannel,paytype,createtime,innerorderid, merchantid ) values ( 'p1514540980419654149196',6.0,1,2,1,'2017-12-29 17:49:40','13de6f7b-70ac-42b8-a4af-90b731ba7f5d',3 ) TABLE LOCK table `wukong`.`pay_record` trx id 653873314 lock mode IX ---TRANSACTION 653872759, ACTIVE 244 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 137700, OS thread handle 140288419092224, query id 2106153054 10.103.22.176 developer update insert into pay_record( orderid,payamount,paystatus,paychannel,paytype,createtime,innerorderid,merchantid ) values ( 'p1514540971143974433048',4.0,1,1,6,'2017-12-29 17:49:31','c5b2f639-c471-43ea-8c2f-51b84e41e30f',102 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872759 lock mode IX ---TRANSACTION 653872744, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 138192, OS thread handle 140288460834560, query id 2106152221 10.103.22.58 developer update insert into pay_record ( orderid, transactionid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540971007823554915', '', 11.0, 1, 9, 1, '2017-12-29 17:49:31', 'bcbda40a-cdde-4166-a2 TABLE LOCK table `wukong`.`pay_record` trx id 653872744 lock mode IX ---TRANSACTION 653872687, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 139043, OS thread handle 140288359913216, query id 2106150823 10.103.22.58 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540970519904371627', 8.0, 1, 2, 1, '2017-12-29 17:49:30', '688b3d32-21ea-45d1-817e-50c1d75d4a7a', 3 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872687 lock mode IX ---TRANSACTION 653872680, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 138486, OS thread handle 140288462948096, query id 2106150519 10.103.22.176 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540970184537333579', 2.0, 1, 2, 4, '2017-12-29 17:49:30', '1861e2f4-af76-4047-ad0c-6b925bee29f6', 5 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872680 lock mode IX ---TRANSACTION 653872679, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 137891, OS thread handle 140288321341184, query id 2106150470 10.103.22.57 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540970455315059367', 8.0, 1, 2, 1, '2017-12-29 17:49:30', '719f45a2-e3fd-4fbe-994b-8733cdbf6d9e', 3 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872679 lock mode IX ---TRANSACTION 653872670, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 137998, OS thread handle 140299701008128, query id 2106150334 10.103.22.58 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540970070860663861', 5.0, 1, 1, 5, '2017-12-29 17:49:30', '2fb176a5-264e-4c43-8316-e19614ebd711', 1 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872670 lock mode IX ---TRANSACTION 653872663, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 138682, OS thread handle 140288621471488, query id 2106150073 10.103.22.177 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540971379715996538', 22.0, 1, 1, 6, '2017-12-29 17:49:31', '17239b8b-70ca-4d57-889c-d8688e68d164', 102 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872663 lock mode IX ---TRANSACTION 653872643, ACTIVE 245 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 1136, 0 row lock(s) MySQL thread id 139024, OS thread handle 140290300778240, query id 2106149508 10.103.22.177 developer update insert into pay_record ( orderid, payamount, paystatus, paychannel, paytype, createtime, innerorderid, merchantid ) values ( 'p1514540970998175883811', 4.95, 1, 1, 1, '2017-12-29 17:49:31', '4e4f9bfa-85ad-4ce5-a4f6-a1dd2d7786f1', 1 ) TABLE LOCK table `wukong`.`pay_record` trx id 653872643 lock mode IX ---TRANSACTION 653872640, ACTIVE 246 sec updating or deleting mysql tables in use 1, locked 1 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1 MySQL thread id 139041, OS thread handle 140299334051584, query id 2106149369 10.103.22.58 developer updating update pay_record SET transactionid = '4200000010201712293798315868', paystatus = 2, payresult = '{transaction_id=4200000010201712293798315868, nonce_str=1749281337, bank_type=CMB_CREDIT, openid=oMAKuuLxdEmrj6xaPGp6wqxbiM_Q, sign=9B70ECC32FACCA48D38E285DEBCE954D, return_msg=OK, fee_type=CNY, mch_id=1232903702, cash_fee=500, out_trade_no=p1514540968234629961205, cash_fee_type=CNY, appid=wxcc603d9f0d54eaf0, total_fee=500, trade_type=MICROPAY, result_code=SUCCESS, attach=, time_end=20171229174929, is_subscribe=N, return_code=SUCCESS}', TABLE LOCK table `wukong`.`pay_record` trx id 653872640 lock mode IX RECORD LOCKS space id 174 page no 2811610 n bits 472 index orderid of table `wukong`.`pay_record` trx id 653872640 lock_mode X locks rec but not gap RECORD LOCKS space id 174 page no 3343611 n bits 120 index PRIMARY of table `wukong`.`pay_record` trx id 653872640 lock_mode X locks rec but not gap -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (read thread) I/O thread 7 state: waiting for completed aio requests (read thread) I/O thread 8 state: waiting for completed aio requests (read thread) I/O thread 9 state: waiting for completed aio requests (read thread) I/O thread 10 state: waiting for completed aio requests (write thread) I/O thread 11 state: waiting for completed aio requests (write thread) I/O thread 12 state: waiting for completed aio requests (write thread) I/O thread 13 state: waiting for completed aio requests (write thread) I/O thread 14 state: waiting for completed aio requests (write thread) I/O thread 15 state: waiting for completed aio requests (write thread) I/O thread 16 state: waiting for completed aio requests (write thread) I/O thread 17 state: waiting for completed aio requests (write thread) Pending normal aio reads: [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: [0, 0, 0, 0, 0, 0, 0, 0] , ibuf aio reads:, log i/o's:, sync i/o's: Pending flushes (fsync) log: 1; buffer pool: 0 2436565 OS file reads, 88421947 OS file writes, 74848558 OS fsyncs 0.40 reads/s, 16384 avg bytes/read, 23.70 writes/s, 23.70 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 23, seg size 25, 226087 merges merged operations: insert 210357, delete mark 18409, delete 4 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 13279583, node heap has 1515 buffer(s) Hash table size 13279583, node heap has 37580 buffer(s) Hash table size 13279583, node heap has 7606 buffer(s) Hash table size 13279583, node heap has 6926 buffer(s) Hash table size 13279583, node heap has 744 buffer(s) Hash table size 13279583, node heap has 122 buffer(s) Hash table size 13279583, node heap has 2188 buffer(s) Hash table size 13279583, node heap has 71898 buffer(s) 9.45 hash searches/s, 56.60 non-hash searches/s --- LOG --- Log sequence number 425516816321 Log flushed up to 425516815828 Pages flushed up to 424659831447 Last checkpoint at 424659831447 Max checkpoint age 5216728781 Checkpoint age target 5053706007 Modified age 856984874 Checkpoint age 856984874 1 pending log flushes, 0 pending chkp writes 60488384 log i/o's done, 23.70 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 55889100800 Dictionary memory allocated 7304743 Internal hash tables (constant factor + variable factor) Adaptive hash index 2956565312 (849893312 + 2106672000) Page hash 6640792 (buffer pool 0 only) Dictionary cache 219778071 (212473328 + 7304743) File system 1186784 (812272 + 374512) Lock system 148481944 (147837832 + 644112) Recovery system 0 (0 + 0) Buffer pool size 3276400 Buffer pool size, bytes 0 Free buffers 32000 Database pages 3115821 Old database pages 1150012 Modified db pages 238312 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 456215, not young 225944 0.55 youngs/s, 0.25 non-youngs/s Pages read 2436202, created 780241, written 22799024 0.40 reads/s, 0.15 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3115821, unzip_LRU len: 0 I/O sum[64]:cur[8], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389539 Old database pages 143774 Modified db pages 28468 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 50232, not young 25720 0.00 youngs/s, 0.00 non-youngs/s Pages read 299868, created 99489, written 2929759 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389539, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4001 Database pages 389387 Old database pages 143718 Modified db pages 29715 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 56914, not young 22064 0.00 youngs/s, 0.00 non-youngs/s Pages read 303168, created 97934, written 2758927 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389387, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389461 Old database pages 143745 Modified db pages 32173 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 71611, not young 61520 0.00 youngs/s, 0.00 non-youngs/s Pages read 309737, created 99172, written 2774323 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389461, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389480 Old database pages 143752 Modified db pages 30576 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 53213, not young 18810 0.05 youngs/s, 0.00 non-youngs/s Pages read 305200, created 94807, written 3021045 0.05 reads/s, 0.05 creates/s, 0.00 writes/s Buffer pool hit rate 999 / 1000, young-making rate 1 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389480, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389533 Old database pages 143772 Modified db pages 27622 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 63452, not young 27234 0.15 youngs/s, 0.15 non-youngs/s Pages read 303499, created 100596, written 2797416 0.10 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 999 / 1000, young-making rate 2 / 1000 not 2 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389533, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389459 Old database pages 143745 Modified db pages 28849 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 55158, not young 21063 0.00 youngs/s, 0.05 non-youngs/s Pages read 303124, created 96954, written 2866826 0.05 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 999 / 1000, young-making rate 0 / 1000 not 1 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389459, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 4000 Database pages 389480 Old database pages 143752 Modified db pages 30562 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 50492, not young 27796 0.15 youngs/s, 0.05 non-youngs/s Pages read 306209, created 94622, written 2903747 0.10 reads/s, 0.05 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 1 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389480, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 409550 Buffer pool size, bytes 0 Free buffers 3999 Database pages 389482 Old database pages 143754 Modified db pages 30347 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 55143, not young 21737 0.20 youngs/s, 0.00 non-youngs/s Pages read 305397, created 96667, written 2746981 0.10 reads/s, 0.05 creates/s, 0.00 writes/s Buffer pool hit rate 997 / 1000, young-making rate 6 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 389482, unzip_LRU len: 0 I/O sum[8]:cur[1], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 10 read views open inside InnoDB 11 RW transactions active inside InnoDB ---OLDEST VIEW--- Read view low limit trx n:o 653872657 Trx read view will not see trx with id >= 653872657, sees < 653872640 Read view individually stored trx ids: Read view trx id 653872640 Read view trx id 653872643 ----------------- Process ID=549, Main thread ID=140300121433856, state: sleeping Number of rows inserted 63645911, updated 39036020, deleted 13125412, read 355385366190 10.90 inserts/s, 15.50 updates/s, 0.00 deletes/s, 15.50 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ ...... END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream 2017-12-29T18:05:23.578264+08:00 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentiona lly crash the server because it appears to be hung. 2017-12-29 18:05:23 0x7f9a2b867700 InnoDB: Assertion failure in thread 140300131923712 in file ut0ut.cc line 917 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 10:05:23 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster key_buffer_size=33554432 read_buffer_size=8388608 max_used_connections=1001 max_threads=1001 thread_count=581 connection_count=550 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 12335399 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x80000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf83ff5] /usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x7c7c94] /lib64/libpthread.so.0[0x35eb80f790] /lib64/libc.so.6(gsignal+0x35)[0x35eb432625] /lib64/libc.so.6(abort+0x175)[0x35eb433e05] /usr/sbin/mysqld(_Z18ut_print_timestampP8_IO_FILE+0x0)[0x7b3794] /usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0xb3)[0x11f70f3] /usr/sbin/mysqld(srv_error_monitor_thread+0x7d6)[0x11a2e66] /lib64/libpthread.so.0[0x35eb807a51] /lib64/libc.so.6(clone+0x6d)[0x35eb4e896d] You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash.