What I see on the customer's host does not correspond to the bug description. The general difference is suggestion about the cause of S-lock is wrong. I used the following branch for debugging: https://github.com/vlad-lesin/percona-server/tree/lp-1735555-xa-transaction-lock-5.7.20-logging-and-stable-test-cust 1) Here is the information from PFS about the locks: mysql> select * from information_schema.INNODB_LOCKS; +----------------------------+-------------+-----------+-----------+---------------------+------------+------------+-----------+----------+------------------------+ | lock_id | lock_trx_id | lock_mode | lock_type | lock_table | lock_index | lock_space | lock_page | lock_rec | lock_data | +----------------------------+-------------+-----------+-----------+---------------------+------------+------------+-----------+----------+------------------------+ | 28911602068:8421:1056018:1 | 28911602068 | X | RECORD | `XXX`.`YYY` | PRIMARY | 8421 | 1056018 | 1 | supremum pseudo-record | | 28911602059:8421:1056018:1 | 28911602059 | S | RECORD | `XXX`.`YYY` | PRIMARY | 8421 | 1056018 | 1 | supremum pseudo-record | +----------------------------+-------------+-----------+-----------+---------------------+------------+------------+-----------+----------+------------------------+ 2 rows in set, 1 warning (0.01 sec) Pay attention to transaction id's. 28911602068 - for S-lock 28911602059 - for X-lock 2) The following is the backtrace of the S-lock: =========bt 2.1=============== Breakpoint 1, RecLock::init (this=this@entry=0x7ff49c058af0, page=) at ./storage/innobase/include/lock0priv.h:863 863 ib::info() << "^^^^^^^^^^^^^^^^^^SET BREAKPOINT HERE!"; Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 nss-softokn-freebl-3.28.3-8.el7_4.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) p m_trx $1 = (trx_t *) 0x0 (gdb) bt #0 RecLock::init (this=this@entry=0x7ff49c058af0, page=) at ./storage/innobase/include/lock0priv.h:863 #1 0x0000000000f3c5fd in RecLock (mode=, heap_no=140724951089696, block=0x7ffd14b88220, index=0x7ff40cb5e858, this=0x7ff49c058af0) at ./storage/innobase/include/lock0priv.h:697 #2 lock_rec_add_to_queue (type_mode=, block=block@entry=0x7ffd14b88220, heap_no=heap_no@entry=1, index=0x7ff40cb5e858, trx=0x7fffea8650c8, caller_owns_trx_mutex=false) at ./storage/innobase/lock/lock0lock.cc:1880 #3 0x0000000000f3c790 in lock_rec_inherit_to_gap (heir_block=heir_block@entry=0x7ffd14b88220, block=block@entry=0x7ffd14b91fa0, heir_heap_no=heir_heap_no@entry=1, heap_no=heap_no@entry=2) at ./storage/innobase/lock/lock0lock.cc:2744 #4 0x0000000000f3cf50 in lock_update_split_right (right_block=right_block@entry=0x7ffd14b91fa0, left_block=left_block@entry=0x7ffd14b88220) at ./storage/innobase/lock/lock0lock.cc:3318 #5 0x00000000010a10e3 in btr_page_split_and_insert (flags=flags@entry=0, cursor=cursor@entry=0x7ff49c059070, offsets=offsets@entry=0x7ff49c058fd0, heap=heap@entry=0x7ff49c058fc0, tuple=tuple@entry=0x7ff40d196c58, n_ext=, mtr=mtr@entry=0x7ff49c0597b0) at ./storage/innobase/btr/btr0btr.cc:2846 #6 0x00000000010af38a in btr_cur_pessimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7ff49c059070, offsets=offsets@entry=0x7ff49c058fd0, heap=heap@entry=0x7ff49c058fc0, entry=entry@entry=0x7ff40d196c58, rec=rec@entry=0x7ff49c059490, big_rec=big_rec@entry=0x7ff49c058fb0, n_ext=n_ext@entry=0, thr=thr@entry=0x7ff40d1951e8, mtr=mtr@entry=0x7ff49c0597b0) at ./storage/innobase/btr/btr0cur.cc:3497 #7 0x0000000000fcb442 in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=, mode@entry=33, index=index@entry=0x7ff40cb5e858, n_uniq=n_uniq@entry=1, entry=entry@entry=0x7ff40d196c58, n_ext=n_ext@entry=0, thr=thr@entry=0x7ff40d1951e8, dup_chk_only=dup_chk_only@entry=false) at ./storage/innobase/row/row0ins.cc:2655 #8 0x0000000000fcf6f7 in row_ins_clust_index_entry (index=0x7ff40cb5e858, entry=0x7ff40d196c58, thr=thr@entry=0x7ff40d1951e8, n_ext=n_ext@entry=0, dup_chk_only=dup_chk_only@entry=false) at ./storage/innobase/row/row0ins.cc:3397 #9 0x0000000000fd0c28 in row_ins_index_entry (thr=0x7ff40d1951e8, entry=, index=) at ./storage/innobase/row/row0ins.cc:3513 #10 row_ins_index_entry_step (thr=0x7ff40d1951e8, node=0x7ff40cb56310) at ./storage/innobase/row/row0ins.cc:3665 #11 row_ins (thr=0x7ff49c059f00, node=) at ./storage/innobase/row/row0ins.cc:3811 #12 row_ins_step (thr=thr@entry=0x7ff40d1951e8) at ./storage/innobase/row/row0ins.cc:4003 #13 0x0000000000fe357b in row_insert_for_mysql_using_ins_graph (mysql_rec=mysql_rec@entry=0x7ff40cb4a7c0 "", prebuilt=prebuilt@entry=0x7ff40cb555c8) at ./storage/innobase/row/row0mysql.cc:2279 #14 0x0000000000fe72d4 in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7ff40cb4a7c0 "", prebuilt=prebuilt@entry=0x7ff40cb555c8) at ./storage/innobase/row/row0mysql.cc:2403 #15 0x0000000000ef7ac1 in ha_innobase::write_row (this=0x7ff40cb4a280, record=) at ./storage/innobase/handler/ha_innodb.cc:8315 #16 0x00000000007ff122 in handler::ha_write_row (this=0x7ff40cb4a280, buf=0x7ff40cb4a7c0 "") at ./sql/handler.cc:8437 #17 0x0000000000e3101d in Write_rows_log_event::write_row (this=this@entry=0x7ff40d2c28b0, rli=, overwrite=false) at ./sql/log_event.cc:12650 #18 0x0000000000e31349 in Write_rows_log_event::do_exec_row (this=0x7ff40d2c28b0, rli=) at ./sql/log_event.cc:12850 #19 0x0000000000e1e256 in Rows_log_event::do_apply_row (this=this@entry=0x7ff40d2c28b0, rli=rli@entry=0x3e92dc0) at ./sql/log_event.cc:10206 #20 0x0000000000e2f336 in Rows_log_event::do_apply_event (this=0x7ff40d2c28b0, rli=0x3e92dc0) at ./sql/log_event.cc:11385 #21 0x0000000000e27b62 in Log_event::apply_event (this=this@entry=0x7ff40d2c28b0, rli=rli@entry=0x3e92dc0) at ./sql/log_event.cc:3411 #22 0x0000000000e6d251 in apply_event_and_update_pos (ptr_ev=ptr_ev@entry=0x7ff49c05a8d0, thd=thd@entry=0x7ff40c000910, rli=rli@entry=0x3e92dc0) at ./sql/rpl_slave.cc:4782 #23 0x0000000000e795c2 in exec_relay_log_event (rli=0x3e92dc0, thd=0x7ff40c000910) at ./sql/rpl_slave.cc:5306 #24 handle_slave_sql (arg=arg@entry=0x3dd7760) at ./sql/rpl_slave.cc:7541 #25 0x0000000001228974 in pfs_spawn_thread (arg=0x7ff4819fce40) at ./storage/perfschema/pfs.cc:2190 #26 0x00007ffff7bc6e25 in start_thread () from /lib64/libpthread.so.0 #27 0x00007ffff5fa634d in clone () from /lib64/libc.so.6 (gdb) fr 2 #2 lock_rec_add_to_queue (type_mode=, block=block@entry=0x7ffd14b88220, heap_no=heap_no@entry=1, index=0x7ff40cb5e858, trx=0x7fffea8650c8, caller_owns_trx_mutex=false) at ./storage/innobase/lock/lock0lock.cc:1880 1880 RecLock rec_lock(index, block, heap_no, type_mode); (gdb) p trx->id $2 = 28911602059 ==================================== Please, pay attention to transaction id: 28911602059. It corresponds to S-lock transaction id from PFS in (1). If we look at the backtrace we will see that the S-lock is caused by innodb page split process. I added some additional logging to error log to debug. And the XA which causes the S-lock is the following: XA START X'46966a009c509b3869702d31302d3230352d322d37322c7365727665722c503337323331',X'69702d31302d3230352d322d37322c7365727665722c5033373233312c00',4871251 So if we have query log, we can find the transaction. 3) The X-lock backtrace is the following: ==========bt 3.1=================== Breakpoint 1, RecLock::init (this=this@entry=0x7ff49c058c70, page=) at ./storage/innobase/include/lock0priv.h:863 863 ib::info() << "^^^^^^^^^^^^^^^^^^SET BREAKPOINT HERE!"; (gdb) bt #0 RecLock::init (this=this@entry=0x7ff49c058c70, page=) at ./storage/innobase/include/lock0priv.h:863 #1 0x0000000000f4543f in RecLock (prdt=0x0, mode=2563, heap_no=1, block=0x7ffd14b88220, index=0x7ff40cb5e858, thr=0x1, this=0x7ff49c058c70) at ./storage/innobase/include/lock0priv.h:656 #2 lock_rec_insert_check_and_lock (flags=flags@entry=0, rec=, block=0x7ffd14b88220, index=index@entry=0x7ff40cb5e858, thr=thr@entry=0x7ff40d1951e8, mtr=mtr@entry=0x7ff49c0597b0, inherit=inherit@entry=0x7ff49c058eb0) at ./storage/innobase/lock/lock0lock.cc:5969 #3 0x00000000010af519 in btr_cur_ins_lock_and_undo (inherit=0x7ff49c058eb0, mtr=0x7ff49c0597b0, mtr@entry=0x7ff40cb5e858, thr=0x7ff40d1951e8, thr@entry=0x7ff49c058fb0, entry=0x7ff40d196c58, cursor=0x7ff49c059070, flags=) at ./storage/innobase/btr/btr0cur.cc:3035 #4 btr_cur_pessimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7ff49c059070, offsets=offsets@entry=0x7ff49c058fd0, heap=heap@entry=0x7ff49c058fc0, entry=entry@entry=0x7ff40d196c58, rec=rec@entry=0x7ff49c059490, big_rec=big_rec@entry=0x7ff49c058fb0, n_ext=n_ext@entry=0, thr=thr@entry=0x7ff40d1951e8, mtr=mtr@entry=0x7ff49c0597b0) at ./storage/innobase/btr/btr0cur.cc:3438 #5 0x0000000000fcb442 in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=, mode@entry=33, index=index@entry=0x7ff40cb5e858, n_uniq=n_uniq@entry=1, entry=entry@entry=0x7ff40d196c58, n_ext=n_ext@entry=0, thr=thr@entry=0x7ff40d1951e8, dup_chk_only=dup_chk_only@entry=false) at ./storage/innobase/row/row0ins.cc:2655 #6 0x0000000000fcf6f7 in row_ins_clust_index_entry (index=0x7ff40cb5e858, entry=0x7ff40d196c58, thr=thr@entry=0x7ff40d1951e8, n_ext=n_ext@entry=0, dup_chk_only=dup_chk_only@entry=false) at ./storage/innobase/row/row0ins.cc:3397 #7 0x0000000000fd0c28 in row_ins_index_entry (thr=0x7ff40d1951e8, entry=, index=) at ./storage/innobase/row/row0ins.cc:3513 #8 row_ins_index_entry_step (thr=0x7ff40d1951e8, node=0x7ff40cb56310) at ./storage/innobase/row/row0ins.cc:3665 #9 row_ins (thr=0x7ff49c059f00, node=) at ./storage/innobase/row/row0ins.cc:3811 #10 row_ins_step (thr=thr@entry=0x7ff40d1951e8) at ./storage/innobase/row/row0ins.cc:4003 #11 0x0000000000fe357b in row_insert_for_mysql_using_ins_graph (mysql_rec=mysql_rec@entry=0x7ff40cb4a7c0 "", prebuilt=prebuilt@entry=0x7ff40cb555c8) at ./storage/innobase/row/row0mysql.cc:2279 #12 0x0000000000fe72d4 in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7ff40cb4a7c0 "", prebuilt=prebuilt@entry=0x7ff40cb555c8) at ./storage/innobase/row/row0mysql.cc:2403 #13 0x0000000000ef7ac1 in ha_innobase::write_row (this=0x7ff40cb4a280, record=) at ./storage/innobase/handler/ha_innodb.cc:8315 #14 0x00000000007ff122 in handler::ha_write_row (this=0x7ff40cb4a280, buf=0x7ff40cb4a7c0 "") at ./sql/handler.cc:8437 #15 0x0000000000e3101d in Write_rows_log_event::write_row (this=this@entry=0x7ff40cb0d410, rli=, overwrite=false) at ./sql/log_event.cc:12650 #16 0x0000000000e31349 in Write_rows_log_event::do_exec_row (this=0x7ff40cb0d410, rli=) at ./sql/log_event.cc:12850 #17 0x0000000000e1e256 in Rows_log_event::do_apply_row (this=this@entry=0x7ff40cb0d410, rli=rli@entry=0x3e92dc0) at ./sql/log_event.cc:10206 #18 0x0000000000e2f336 in Rows_log_event::do_apply_event (this=0x7ff40cb0d410, rli=0x3e92dc0) at ./sql/log_event.cc:11385 #19 0x0000000000e27b62 in Log_event::apply_event (this=this@entry=0x7ff40cb0d410, rli=rli@entry=0x3e92dc0) at ./sql/log_event.cc:3411 #20 0x0000000000e6d251 in apply_event_and_update_pos (ptr_ev=ptr_ev@entry=0x7ff49c05a8d0, thd=thd@entry=0x7ff40c000910, rli=rli@entry=0x3e92dc0) at ./sql/rpl_slave.cc:4782 #21 0x0000000000e795c2 in exec_relay_log_event (rli=0x3e92dc0, thd=0x7ff40c000910) at ./sql/rpl_slave.cc:5306 #22 handle_slave_sql (arg=arg@entry=0x3dd7760) at ./sql/rpl_slave.cc:7541 #23 0x0000000001228974 in pfs_spawn_thread (arg=0x7ff4819fce40) at ./storage/perfschema/pfs.cc:2190 #24 0x00007ffff7bc6e25 in start_thread () from /lib64/libpthread.so.0 #25 0x00007ffff5fa634d in clone () from /lib64/libc.so.6 (gdb) p m_trx->id $5 = 28911602068 ================================== Pay attention to transaction id 28911602068 which corresponds to X-lock in the table in (1). I don't see any special things here, just usual Write_rows_log_event. It's XA transaction id is the following: XA START X'5d966a009c509b3869702d31302d3230352d322d37322c7365727665722c503337323331',X'69702d31302d3230352d322d37322c7365727665722c5033373233312c00',4871251 Summing up: The S-lock on the customer's host is caused by page split process, but not foreign key checking as described in the bug report. So the initial test case is useless for us. The whole picture is the following: 1) XA 1 contains queries which cause page split which, in turn, causes the supremum S-lock; 2) While XA 1 is not committed and the supremum S-lock is not released the transaction XA 2 starts and tries to modify the page S-locked by XA 1, what causes the supremum X-lock, XA 2 is pushed to waiting queue; 3) Lock wait timeout errors, the slave thread stops.