SQL statement missing in SHOW ENGINE INNODB STATUS

Bug #1328824 reported by Peiran Song
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Laurynas Biveinis
5.1
Invalid
Undecided
Unassigned
5.5
Invalid
Undecided
Unassigned
5.6
Fix Released
High
Laurynas Biveinis

Bug Description

Observed in Percona Server 5.6.16 and 5.6.17 that SQL statement sometime is missing in SHOW ENGINE INNODB STATUS output both in the LASTEST DETECTED DEADLOCK section and TRANSACTIONS section. Stock 5.6 and PS 5.5 don't seem affected, but testing was limited.

This is a customer's data with PS 5.6.17. Autocommit is on, and TRANSACTION1's SQL statement is not printed.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-06-10 17:01:36 7f4afe16d700
*** (1) TRANSACTION:
TRANSACTION 505926011, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
LOCK WAIT 4 lock struct(s), heap size 1184, 10 row lock(s)
MySQL thread id 34797, OS thread handle 0x7f4af79a6700, query id 55052467 10.1.1.111 recall Sending data
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1373356 page no 552596 n bits 112 index `PRIMARY` of table `outcalls`.`outcalls_complete` trx id 505926011 lock mode S locks rec but not gap waiting
*** (2) TRANSACTION:
TRANSACTION 505925999, ACTIVE 0 sec fetching rows
mysql tables in use 1, locked 1
133 lock struct(s), heap size 13864, 1519 row lock(s)
MySQL thread id 83098, OS thread handle 0x7f4afe16d700, query id 55052397 10.1.9.29 recall Searching rows for update
UPDATE outcalls_complete set status = status-8000 where status >= 9000 AND node = NAME_CONST('nClusterNode',5093)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1373356 page no 552596 n bits 112 index `PRIMARY` of table `outcalls`.`outcalls_complete` trx id 505925999 lock_mode X locks rec but not gap
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1373356 page no 552596 n bits 112 index `PRIMARY` of table `outcalls`.`outcalls_complete` trx id 505925999 lock_mode X locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (1)

I produced a deadlock on PS 5.6.16 and noticed the same thing. Two running SQLs, but statement from TRANSACTION1 is missing.
------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-06-10 10:01:18 7f81ccac7700
*** (1) TRANSACTION:
TRANSACTION 10994, ACTIVE 3 sec setting auto-inc lock
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 108, OS thread handle 0x7f81ccb08700, query id 448 localhost 127.0.0.1 msandbox update
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `test`.`child` trx id 10994 lock mode AUTO-INC waiting
*** (2) TRANSACTION:
TRANSACTION 10993, ACTIVE 7 sec inserting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1184, 1 row lock(s), undo log entries 1
MySQL thread id 132, OS thread handle 0x7f81ccac7700, query id 445 localhost 127.0.0.1 msandbox update
insert into child (p_id) values (2)
*** (2) HOLDS THE LOCK(S):
TABLE LOCK table `test`.`child` trx id 10993 lock mode AUTO-INC
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 21 page no 3 n bits 72 index `PRIMARY` of table `test`.`parent` trx id 10993 lock mode S locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (1)

More testing showed that this is not limited to transactions involved in deadlock. With a large table, running an UPDATE statement, and meanwhile, look at the SHOW ENGINE INNODB STATUS, the running statement is not printed in the TRANSACTION section.

mysql> update t1 set name = 'ah';

---TRANSACTION 11305, ACTIVE 1 sec fetching rows
mysql tables in use 1, locked 1
2855 lock struct(s), heap size 357928, 791828 row lock(s), undo log entries 788975
MySQL thread id 202, OS thread handle 0x7f81ccb49700, query id 790 localhost 127.0.0.1 msandbox updating
TABLE LOCK table `test`.`t1` trx id 11305 lock mode IX
RECORD LOCKS space id 34 page no 5 n bits 528 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 6 n bits 360 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 3776 n bits 344 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 7 n bits 360 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 3840 n bits 344 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 10 n bits 360 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 3904 n bits 344 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 12 n bits 360 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
RECORD LOCKS space id 34 page no 384 n bits 344 index `PRIMARY` of table `test`.`t1` trx id 11305 lock_mode X
TOO MANY LOCKS PRINTED FOR THIS TRX: SUPPRESSING FURTHER PRINTS

tags: added: i42931
Revision history for this message
Peiran Song (peiran-song) wrote :
Download full text (3.7 KiB)

Again limited test on PS 5.1 and PS 5.5 didn't reproduce it.

In PS 5.6, a test case to produce a deadlock with running statements and observe missing statement in deadlock section :

Prepare the data:
-------------------------
use test;
create table parent (id int, name varchar(20), primary key (id)) engine = innodb;
create table child (id int auto_increment, p_id int, primary key (id), key idx_p_id (p_id), constraint fk_p_id foreign key (p_id) references parent (id) ) engine = innodb;
insert into parent values (1, "foo"), (2,"blah");
insert into child (p_id) values (1);

for i in {1..20}; do echo $i; echo "insert into child (p_id) select t.p_id from child t" | mysql test; done

Run the test:
-------------------
session1:
echo “insert into child (p_id) select t.p_id from child t” | mysql test ;

session2: while session1 running
mysql> insert into child (p_id) value (2);

session3, while session1 and session2 running
mysql> begin; update parent set name = “aha” where id = 2; insert into child (p_id) value (1);

When session1 finishes, session2 and 3 would result in deadlock.

mysql> show engine innodb status\G

------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-06-12 11:44:45 7f81ccb8a700
*** (1) TRANSACTION:
TRANSACTION 12040, ACTIVE 4 sec setting auto-inc lock
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 279, OS thread handle 0x7f81ccac7700, query id 1244 localhost 127.0.0.1 msandbox update
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `test`.`child` trx id 12040 lock mode AUTO-INC waiting
*** (2) TRANSACTION:
TRANSACTION 12039, ACTIVE 7 sec inserting
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1184, 1 row lock(s), undo log entries 1
MySQL thread id 280, OS thread handle 0x7f81ccb8a700, query id 1241 localhost 127.0.0.1 msandbox update
insert into child (p_id) value (2)
*** (2) HOLDS THE LOCK(S):
TABLE LOCK table `test`.`child` trx id 12039 lock mode AUTO-INC
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 21 page no 3 n bits 72 index `PRIMARY` of table `test`.`parent` trx id 12039 lock mode S locks rec but not gap waiting
*** WE ROLL BACK TRANSACTION (1)

To observe missing statement in TRANSACTION section, use the above data:

session1:
mysql > update child set p_id = 2;

session 2 while session 1 running:
mysql> show engine innodb status\G

---TRANSACTION 12058, ACTIVE 2 sec updating or deleting
mysql tables in use 1, locked 1
109 lock struct(s), heap size 30248, 58258 row lock(s), undo log entries 58152
MySQL thread id 280, OS thread handle 0x7f81ccb8a700, query id 1247 localhost 127.0.0.1 msandbox updating
TABLE LOCK table `test`.`child` trx id 12058 lock mode IX
RECORD LOCKS space id 39 page no 5 n bits 624 index `PRIMARY` of table `test`.`child` trx id 12058 lock_mode X
TABLE LOCK table `test`.`parent` trx id 12058 lock mode IS
RECORD LOCKS space id 21 page no 3 n bits 72 index `PRIMARY` of table `test`.`parent` trx id 12058 lock mode S locks rec but not gap
RECORD LOCKS space id 39 page no 6 n bits 624 index `PRIMARY` of table `test`.`child` trx id 12058 lock_mode X
RECORD ...

Read more...

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

"Stock 5.6 don't seem affected" - is that confirmed with the testcase from comment #1?

Revision history for this message
Peiran Song (peiran-song) wrote :

Laurynas,

I couldn't use the above testcase on stock v5.6 due to http://bugs.mysql.com/bug.php?id=72748.

I tested stock v5.6 with a large table where "id" column has data from 1 to 1M, and "a" column has data from 1M to 1, and two sessions scanning and locking the rows from two directions. I tried it again and captured the innodb output:

Server version: 5.6.19 MySQL Community Server (GPL)

------------------------
LATEST DETECTED DEADLOCK
------------------------
2014-06-13 16:27:01 7f191c1f3700
*** (1) TRANSACTION:
TRANSACTION 20495, ACTIVE 12 sec fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 2100 lock struct(s), heap size 357928, 1196493 row lock(s)
MySQL thread id 3, OS thread handle 0x7f191c2b6700, query id 51 localhost root Sending data
select a from t1 order by a for update
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 64 page no 1295 n bits 576 index `PRIMARY` of table `test`.`t1` trx id 20495 lock_mode X locks rec but not gap waiting
Record lock, heap no 349 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
 0: len 4; hex 8009dffd; asc ;;
 1: len 6; hex 00000000500e; asc P ;;
 2: len 7; hex 0a000008f110d9; asc ;;
 3: len 4; hex 80056243; asc bC;;
 4: len 3; hex 666f6f; asc foo;;

*** (2) TRANSACTION:
TRANSACTION 20494, ACTIVE 13 sec fetching rows
mysql tables in use 1, locked 1
1632 lock struct(s), heap size 226856, 452209 row lock(s), undo log entries 450579
MySQL thread id 7, OS thread handle 0x7f191c1f3700, query id 50 localhost root updating
update t1 set b = "foo"
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 64 page no 1295 n bits 576 index `PRIMARY` of table `test`.`t1` trx id 20494 lock_mode X
Record lock, heap no 349 PHYSICAL RECORD: n_fields 5; compact format; info bits 0
 0: len 4; hex 8009dffd; asc ;;
 1: len 6; hex 00000000500e; asc P ;;
 2: len 7; hex 0a000008f110d9; asc ;;
 3: len 4; hex 80056243; asc bC;;
 4: len 3; hex 666f6f; asc foo;;
.....
......
.........

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

Why http://bugs.mysql.com/bug.php?id=72748 is not preventing Percona Server testing?

Revision history for this message
Peiran Song (peiran-song) wrote :

I was able to run the test case on Percona Server a few times, and I was not able to run the test case on Community version because every time the simple insert succeeds while the insert...select... is running.

I observed different behaviors. Regarding why Percona Server and Community version act differently on this, I will leave it to some one else to answer.

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

This might be an upstream bug and we need to find out in order to process this bug correctly.

There was another reproduction scenario described above: "With a large table, running an UPDATE statement, and meanwhile, look at the SHOW ENGINE INNODB STATUS, the running statement is not printed in the TRANSACTION section." that does not depend on insert...select...

Revision history for this message
Peiran Song (peiran-song) wrote :

For this reproducible case, upstream mysql prints the statement as expected. In summary, the test cases mentioned in this bug had been ran on stock 5.6, PS 5.1, PS 5.5 and PS 5.6, and only saw missing statements with PS 5.6.

tags: added: i44350
Revision history for this message
Miguel Angel Nieto (miguelangelnieto) wrote :

More issues being added to the list. We should increase the priority of this bug.

tags: added: i44392
tags: added: i44616
Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

This is a regression caused by the Twitter statement timeout patch. It has been fixed in Twitter branch by https://github.com/twitter/mysql/commit/22262833515e90e90c39d2a012a95da59ce027e1

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

There is also a Facebook fix for the issue that the Twitter statement timeout patch fixed originally (http://bugs.mysql.com/bug.php?id=60682) and that partially fixes the current issue: https://github.com/facebook/mysql-5.6/commit/47ebc7a5e8aad215317e75e7eab077cd3e3e05b9.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-793

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.