There are a bunch of neutron transactions active for 1397 seconds. Some for 96 seconds. And a bunch that are "not started".
Two of them indicate they have been waiting on a lock like so:
---TRANSACTION 57170620, ACTIVE 1397 sec
MySQL thread id 169, OS thread handle 0x7fa3375f5700, query id 36577 localhost 127.0.0.1 neutron
Trx read view will not see trx with id >= 57170621, sees < 57170585
---TRANSACTION 57170619, ACTIVE 1397 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 168, OS thread handle 0x7fa337637700, query id 123513 localhost 127.0.0.1 neutron updating
UPDATE agents SET heartbeat_timestamp='2019-11-19 20:54:45.480969' WHERE agents.id = '1d7c5726-a336-4156-ace5-952d5363ad50'
Trx read view will not see trx with id >= 57170620, sees < 57170585
------- TRX HAS BEEN WAITING 17 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 204 page no 3 n bits 88 index `PRIMARY` of table `neutron`.`agents` trx id 57170619 lock_mode X locks rec but not gap waiting
Record lock, heap no 6 PHYSICAL RECORD: n_fields 16; compact format; info bits 0
0: len 30; hex 31643763353732362d613333362d343135362d616365352d393532643533; asc 1d7c5726-a336-4156-ace5-952d53; (total 36 bytes);
1: len 6; hex 000003685a9e; asc hZ ;;
2: len 7; hex 560000026d1233; asc V m 3;;
3: len 14; hex 4d65746164617461206167656e74; asc Metadata agent;;
4: len 22; hex 6e657574726f6e2d6d657461646174612d6167656e74; asc neutron-metadata-agent;;
5: len 3; hex 4e2f41; asc N/A;;
6: len 30; hex 64333062303339302d323066642d343366652d393566382d313037396330; asc d30b0390-20fd-43fe-95f8-1079c0; (total 36 bytes);
7: len 1; hex 81; asc ;;
8: len 5; hex 99a34f6033; asc O`3;;
9: len 5; hex 99a41b7ab4; asc z ;;
10: len 5; hex 99a4a74dad; asc M ;;
11: SQL NULL;
12: len 30; hex 7b226c6f675f6167656e745f68656172746265617473223a2066616c7365; asc {"log_agent_heartbeats": false; (total 158 bytes);
13: len 4; hex 80000000; asc ;;
14: SQL NULL;
15: SQL NULL;
Finally since i'm not an SQL expert, here is output of SHOW ENGINE INNODB STATUS': http:// paste.openstack .org/show/ 786369/
There are a bunch of neutron transactions active for 1397 seconds. Some for 96 seconds. And a bunch that are "not started".
Two of them indicate they have been waiting on a lock like so:
---TRANSACTION 57170620, ACTIVE 1397 sec timestamp= '2019-11- 19 20:54:45.480969' WHERE agents.id = '1d7c5726- a336-4156- ace5-952d5363ad 50' 62d613333362d34 3135362d6163653 52d393532643533 ; asc 1d7c5726- a336-4156- ace5-952d53; (total 36 bytes); 1206167656e74; asc Metadata agent;; d6d657461646174 612d6167656e74; asc neutron- metadata- agent;; 02d323066642d34 3366652d3935663 82d313037396330 ; asc d30b0390- 20fd-43fe- 95f8-1079c0; (total 36 bytes); 7656e745f686561 727462656174732 23a2066616c7365 ; asc {"log_agent_ heartbeats" : false; (total 158 bytes);
MySQL thread id 169, OS thread handle 0x7fa3375f5700, query id 36577 localhost 127.0.0.1 neutron
Trx read view will not see trx with id >= 57170621, sees < 57170585
---TRANSACTION 57170619, ACTIVE 1397 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 360, 1 row lock(s)
MySQL thread id 168, OS thread handle 0x7fa337637700, query id 123513 localhost 127.0.0.1 neutron updating
UPDATE agents SET heartbeat_
Trx read view will not see trx with id >= 57170620, sees < 57170585
------- TRX HAS BEEN WAITING 17 SEC FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 204 page no 3 n bits 88 index `PRIMARY` of table `neutron`.`agents` trx id 57170619 lock_mode X locks rec but not gap waiting
Record lock, heap no 6 PHYSICAL RECORD: n_fields 16; compact format; info bits 0
0: len 30; hex 316437633537323
1: len 6; hex 000003685a9e; asc hZ ;;
2: len 7; hex 560000026d1233; asc V m 3;;
3: len 14; hex 4d6574616461746
4: len 22; hex 6e657574726f6e2
5: len 3; hex 4e2f41; asc N/A;;
6: len 30; hex 643330623033393
7: len 1; hex 81; asc ;;
8: len 5; hex 99a34f6033; asc O`3;;
9: len 5; hex 99a41b7ab4; asc z ;;
10: len 5; hex 99a4a74dad; asc M ;;
11: SQL NULL;
12: len 30; hex 7b226c6f675f616
13: len 4; hex 80000000; asc ;;
14: SQL NULL;
15: SQL NULL;