node hanging in secondary index test

[Seppo Jaakola]

Reproduced in 2 node multi-master topology with test case:

seppo@pitbull:~/work/sqlgen/trunk/src$ ./sqlgen --host=0 --port=3307 --user=root --password=rootpass --tables=2 --rows=100 --create=1 --duration=0

seppo@pitbull:~/work/sqlgen/trunk/src$ ./sqlgen --host=0 --port=3309 --user=root --password=rootpass --tables=2 --rows=100 --create=0 --duration=600 --secondary=0.5 --users=12;

This issue has been see only with builtin innobase, maybe plugin version is not vulnerable

During the test load, one node can remain hanging in a state where slave applier is waiting for a victim thread to abort lock wait and grant the blocking lock to slave applier.

[Seppo Jaakola (seppo-jaakola)]

Applier:
Thread 7 (Thread 0x7f33bdba0700 (LWP 30721)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1 0x00000000009bc3ff in safe_cond_wait (cond=0x23b5db8, mp=0x23b5d40, file=0xb1d002 "os/os0sync.c", line=422) at thr_mutex.c:237
#2 0x00000000008c747f in os_event_wait_low (event=0x23b5d40, reset_sig_count=0) at os/os0sync.c:422
#3 0x00000000008f089a in srv_suspend_mysql_thread (thr=0x7f33e049a8d8) at srv/srv0srv.c:1567
#4 0x00000000008db22b in row_mysql_handle_errors (new_err=0x7f33bdb9db50, trx=0x7f33e046c8b8, thr=0x7f33e049a8d8, savept=0x7f33bdb9db20) at row/row0mysql.c:497
#5 0x00000000008dc2fc in row_insert_for_mysql (mysql_rec=0x7f33b8037048 "\341\021", prebuilt=0x7f33e04618b8) at row/row0mysql.c:1177
#6 0x00000000008698a6 in ha_innobase::write_row (this=0x7f33b8036e58, record=0x7f33b8037048 "\341\021") at handler/ha_innodb.cc:4473
#7 0x000000000077ae2c in handler::ha_write_row (this=0x7f33b8036e58, buf=0x7f33b8037048 "\341\021") at handler.cc:4722
#8 0x000000000073b94e in Rows_log_event::write_row (this=0x7f33b80d0b68, rli=0x48d8ce0, overwrite=false) at log_event.cc:8675
#9 0x000000000073baa7 in Write_rows_log_event::do_exec_row (this=0x7f33b80d0b68, rli=0x48d8ce0) at log_event.cc:8846
#10 0x0000000000737326 in Rows_log_event::do_apply_event (this=0x7f33b80d0b68, rli=0x48d8ce0) at log_event.cc:7617
#11 0x0000000000639e50 in Log_event::apply_event (this=0x7f33b80d0b68, rli=0x48d8ce0) at log_event.h:1067
#12 0x0000000000637c9e in wsrep_bf_apply_rbr (thd=0x48c73e8, rbr_buf=0x7f33b00ae000 "\a,)N\023", buf_len=0) at sql_parse.cc:8420
#13 0x0000000000638286 in wsrep_bf_apply_cb (ctx=0x48c73e8, data=0x7f33bdb9ea80, global_seqno=986647) at sql_parse.cc:8502
#14 0x00007f33bfa39db6 in ?? () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#15 0x00007f33bfa3b951 in ?? () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#16 0x00007f33bfa3be5e in galera::ReplicatorSMM::apply_trx(void*, galera::TrxHandle*) () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#17 0x00007f33bfa41440 in galera::ReplicatorSMM::process_trx(void*, galera::TrxHandle*) () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#18 0x00007f33bfa37c23 in galera::GcsActionSource::dispatch(void*, void const*, unsigned long, gcs_act_type, long, long) () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#19 0x00007f33bfa38091 in galera::GcsActionSource::process(void*) () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#20 0x00007f33bfa3cf50 in galera::ReplicatorSMM::async_recv(void*) () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#21 0x00007f33bfa4ecc3 in galera_recv () from /home/seppo/work/testing/node2/galera/lib/libgalera_smm.so
#22 0x00000000006386fc in wsrep_replication_process (thd=0x48c73e8) at sql_parse.cc:8619
#23 0x00000000006153f8 in start_wsrep_THD (arg=0x638632) at mysqld.cc:4484
#24 0x00007f33e2b20971 in start_thread (arg=<value optimized out>) at pthread_create.c:304
#25 0x00007f33e1d8a92d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#26 0x0000000000000000 in ?? ()

Victim:
#0 pth...