server hangs in shutdown after failed trx replay

[Teemu Ollakka]

Server shutdown hangs in mutex lock after failed transaction replay:

#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
#1 0x00007f1afd9e209b in _L_lock_1006 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f1afd9e201c in __pthread_mutex_lock (mutex=0x4953a58) at pthread_mutex_lock.c:101
#3 0x0000000000519076 in inline_mysql_mutex_lock (src_line=4519, that=0x4953a58, src_file=<optimized out>)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/include/mysql/psi/mysql_thread.h:617
#4 is_replaying_connection (thd=<optimized out>) at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/mysqld.cc:4519
#5 wsrep_close_client_connections (wait_to_end=0 '\000')
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/mysqld.cc:4650
#6 0x0000000000519377 in unireg_abort (exit_code=1) at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/mysqld.cc:1486
#7 0x000000000059a05d in wsrep_mysql_parse (thd=0x49521a0,
    rawbuf=0x7f1ab8014430 "DELETE FROM profile_values WHERE fid=10 AND uid=10", length=50, parser_state=0x7f1ae8100210)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/sql_parse.cc:6028
#8 0x000000000059b9c7 in dispatch_command (command=COM_QUERY, thd=0x49521a0, packet=<optimized out>, packet_length=3893363448)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/sql_parse.cc:1212
#9 0x000000000059bfdd in do_command (thd=0x49521a0)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/sql_parse.cc:869
#10 0x000000000063db95 in do_handle_one_connection (thd_arg=<optimized out>)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/sql_connect.cc:878
#11 0x000000000063ddac in handle_one_connection (arg=0x49521a0)
    at /home/teemu/codership/galera/bzr/codership-mysql/5.5-23/sql/sql_connect.cc:790
#12 0x00007f1afd9dfe9a in start_thread (arg=0x7f1ae8101700) at pthread_create.c:308
#13 0x00007f1afd70ccbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#14 0x0000000000000000 in ?? ()

The reason is that thread calls unireg_abort() with thd->LOCK_wsrep_thd locked and is_replaying_connection() tries to lock the same mutex.