Changing wsrep_slave_threads causes future connections to hang

[Philip Stoev]

Changing the wsrep_slave_threads variable back to the default of 1 causes the SQL connection to hang on subsequent DDL, with the thread blocked as follows:

Thread 5 (Thread 0x7f48dc3ce700 (LWP 34185)):
#0 0x0000003f2e00bcf0 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a78ef in inline_mysql_cond_wait (src_file=0xc056c8 "/home/philips/git/codership-mysql/sql/mysqld.cc", src_line=2787, mutex=0x138bea0 <LOCK_thread_count>, that=0x138ee40 <COND_thread_cache>) at /home/philips/git/codership-mysql/include/mysql/psi/mysql_thread.h:1162
#2 block_until_new_connection () at /home/philips/git/codership-mysql/sql/mysqld.cc:2787
#3 one_thread_per_connection_end (thd=<optimized out>, block_pthread=true) at /home/philips/git/codership-mysql/sql/mysqld.cc:2875
#4 0x00000000006e19b8 in do_handle_one_connection (thd_arg=thd_arg@entry=0x313b9d0) at /home/philips/git/codership-mysql/sql/sql_connect.cc:1026
#5 0x00000000006e1c09 in handle_one_connection (arg=0x313b9d0) at /home/philips/git/codership-mysql/sql/sql_connect.cc:916
#6 0x0000003f2e007f35 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003f2d8f4c3d in clone () from /lib64/libc.so.6

Test case:

--source include/galera_cluster.inc
--source include/have_innodb.inc
CREATE TABLE t1 (f1 INTEGER AUTO_INCREMENT PRIMARY KEY, f2 INTEGER) Engine=InnoDB;
--connection node_2
SET GLOBAL wsrep_slave_threads = 4;
SET GLOBAL wsrep_slave_threads = 1;
DROP TABLE t1;

[Philip Stoev (philip-stoev-f)]

What happens is that, at the end of the test, mysqltest attempts to connect to node_2 in order to check for warnings in the server log. This connection hangs with:

#0 0x0000003f2d8ea7b0 in __poll_nocancel () from /lib64/libc.so.6
#1 0x00000000004493a3 in vio_io_wait (vio=<optimized out>, event=<optimized out>, timeout=120000) at /home/philips/git/codership-mysql/vio/viosocket.c:771
#2 0x0000000000422a53 in mysql_real_connect (mysql=0x7fb1b5ae1010, host=0x4bdec0 "localhost", user=<optimized out>, passwd=0xb27620 "", db=0x0, port=13003, unix_socket=0xb288d1 "/home/philips/git/codership-mysql/mysql-test/var/tmp/mysqld.2.sock", client_flag=2147549184) at /home/philips/git/codership-mysql/sql-common/client.c:3612
#3 0x00000000004102d4 in safe_connect (mysql=mysql@entry=0x7fb1b5ae1010, name=0xb27bd0 "default", host=0xb2891f "localhost", user=0xb288bf "root", pass=0xb27620 "", db=0x0, port=13003, sock=0xb288d1 "/home/philips/git/codership-mysql/mysql-test/var/tmp/mysqld.2.sock") at /home/philips/git/codership-mysql/client/mysqltest.cc:5319
#4 0x000000000040a5ae in main (argc=4, argv=0x7fffd582e3c8) at /home/philips/git/codership-mysql/client/mysqltest.cc:8748

[Philip Stoev (philip-stoev-f)]

To reproduce, put the test case in mysql-test/suite/galera/t and run it:

export WSREP_PROVIDER=...
export PATH=$PATH:/path/to/codership-mysql/scripts/
perl mysql-test-run.pl --suite galera ...

[Seppo Jaakola (seppo-jaakola)]

Terminating slave threads are supposed to be re-used by mysql, and to be handed for new connections. But this does not happen, for some reason, new connections will not get new pthread, but is pending to wait for a donated old slave thread.

[Seppo Jaakola (seppo-jaakola)]

Pushed a fix in revision:
http://bazaar.launchpad.net/~codership/codership-mysql/5.6/revision/4141

The fix skips thread re-use for wsrep applier threads

[Seppo Jaakola (seppo-jaakola)]

The problem did not reproduce on wsrep-5.5 build

[Seppo Jaakola (seppo-jaakola)]

As a clarification for the bug description: not all future connections will hang, just one or a few first connections, depending on how many slave threads were dropped