MySQL do not stop because slave thread hunging

[Marco Tusa]

In a 5 nodes MySQL cluster using galera, with no activity MySQL remain hung and is not able to close clean a node when doing /etc/init.d/mysqld stop.

Checking the log it shows that is waiting to close and additional thread.

Below full information attached as requested by Alex in the email exchange.

130606 16:53:14 [Note] /usr/local/mysql/bin/mysqld: Normal shutdown

130606 16:53:14 [Note] WSREP: Stop replication
130606 16:53:14 [Note] WSREP: Provider disconnect
130606 16:53:14 [Note] WSREP: Closing send monitor...
130606 16:53:14 [Note] WSREP: Closed send monitor.
130606 16:53:14 [Note] WSREP: gcomm: terminating thread
130606 16:53:14 [Note] WSREP: gcomm: joining thread
130606 16:53:14 [Note] WSREP: Before Lock_thread_count
130606 16:53:14 [Note] WSREP: gcomm: closing backend
130606 16:53:15 [Note] WSREP: (5a4429b7-cee9-11e2-0800-722b31e72555, 'tcp://0.0.0.0:4567') turning message relay requesting off
130606 16:53:15 [Warning] WSREP: last inactive check more than PT1.5S ago, skipping check
130606 16:53:15 [Note] WSREP: view(view_id(NON_PRIM,5a4429b7-cee9-11e2-0800-722b31e72555,9) memb {
 5a4429b7-cee9-11e2-0800-722b31e72555,
} joined {
} left {
} partitioned {
 723e50fd-cec8-11e2-0800-672e9e87f901,
 8049192d-cee1-11e2-0800-8a871776ce2d,
 abd23113-cec8-11e2-0800-4b1406d6a7dc,
 ce08c7b4-cec7-11e2-0800-b441f087e5ff,
})
130606 16:53:15 [Note] WSREP: view((empty))
130606 16:53:15 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130606 16:53:15 [Note] WSREP: gcomm: closed
130606 16:53:15 [Note] WSREP: Flow-control interval: [16, 16]
130606 16:53:15 [Note] WSREP: Received NON-PRIMARY.
130606 16:53:15 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8)
130606 16:53:15 [Note] WSREP: Received self-leave message.
130606 16:53:15 [Note] WSREP: Flow-control interval: [0, 0]
130606 16:53:15 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130606 16:53:15 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 8)
130606 16:53:15 [Note] WSREP: RECV thread exiting 0: Success
130606 16:53:15 [Note] WSREP: New cluster view: global state: 727b2c93-cec1-11e2-0800-4453b97f5bac:8, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130606 16:53:15 [Note] WSREP: Setting wsrep_ready to 0
130606 16:53:15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130606 16:53:15 [Note] WSREP: New cluster view: global state: 727b2c93-cec1-11e2-0800-4453b97f5bac:8, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130606 16:53:15 [Note] WSREP: Setting wsrep_ready to 0
130606 16:53:15 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130606 16:53:15 [Note] WSREP: recv_thread() joined.
130606 16:53:15 [Note] WSREP: Closing slave action queue.
130606 16:53:15 [Note] WSREP: applier thread exiting (code:0)
130606 16:53:15 [Note] WSREP: closing applier 1
130606 16:53:17 [Note] WSREP: waiting for client connections to close: 3

..... Hanging ....

[Marco Tusa (marcotusa)]

Forgot to mention this happen with the Percona version AND using the code from codership directly.

the infor reported are from :
| version | 5.5.30-23.7.4-log |
| version_comment | Percona XtraDB Cluster (GPL) 5.5.30-23.7.4, Revision 405, wsrep_23.7.4.r3843

[Alex Yurchenko (ayurchen)]

Marco, what about SHOW FULL PROCESSLIST, SHOW GLOBAL STATUS and SHOW GLOBAL VARIABLES\G?

Specifically, how your configuration is different from defaults? And since it looks like there is some race, what hardware and distribution are you using?

[Marco Tusa (marcotusa)]

weird was not in the file loaded ...
here we go ... the other file

[Marco Tusa (marcotusa)]

ok is my fault ... :o( tooooo tired after a 18hrs spent in front of the console I mix the file ... this one should be the good one sorry about that ... I am just tired.

[Marco Tusa (marcotusa)]

Linux distribution is CentOS 6 64bit .

Running on VM

[Alex Yurchenko (ayurchen)]

Potential fix committed in r152