PXC nodes hang on shutdown when using pool-of-threads

Bug #1185523 reported by Jay Janssen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
High
Raghavendra D Prabhu
5.6
Fix Released
Undecided
Unassigned

Bug Description

Having thread pool enabled:

thread_handling = pool-of-threads

results in PXC not being able to shutdown gracefully.

Normal shutdown looks like this:
130529 13:26:36 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1)
130529 13:26:36 [Note] WSREP: Received self-leave message.
130529 13:26:36 [Note] WSREP: Flow-control interval: [0, 0]
130529 13:26:36 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130529 13:26:36 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1)
130529 13:26:36 [Note] WSREP: RECV thread exiting 0: Success
130529 13:26:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130529 13:26:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:26:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130529 13:26:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:26:36 [Note] WSREP: applier thread exiting (code:0)
130529 13:26:36 [Note] WSREP: recv_thread() joined.
130529 13:26:36 [Note] WSREP: Closing slave action queue.
130529 13:26:36 [Note] WSREP: applier thread exiting (code:5)
130529 13:26:38 [Note] WSREP: rollbacker thread exiting
130529 13:26:38 [Note] Event Scheduler: Purging the queue. 0 events
130529 13:26:38 [Note] WSREP: dtor state: CLOSED
130529 13:26:38 [Note] WSREP: apply mon: entered 0
130529 13:26:38 [Note] WSREP: apply mon: entered 0
130529 13:26:38 [Note] WSREP: mon: entered 4 oooe fraction 0 oool fraction 0
130529 13:26:38 [Note] WSREP: cert index usage at exit 0
130529 13:26:38 [Note] WSREP: cert trx map usage at exit 0
130529 13:26:38 [Note] WSREP: deps set usage at exit 0
130529 13:26:38 [Note] WSREP: avg deps dist 0
130529 13:26:38 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130529 13:26:38 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 1)
130529 13:26:38 [Note] WSREP: Flushing memory map to disk...
130529 13:26:38 InnoDB: Starting shutdown...
130529 13:26:42 InnoDB: Shutdown completed; log sequence number 1598630
130529 13:26:42 [Note] /usr/sbin/mysqld: Shutdown complete

130529 13:26:42 mysqld_safe mysqld from pid file /var/lib/mysql/node3.pid ended

Shutdown with thread pool enabled looks like this:

130529 13:27:36 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1)
130529 13:27:36 [Note] WSREP: Received self-leave message.
130529 13:27:36 [Note] WSREP: Flow-control interval: [0, 0]
130529 13:27:36 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130529 13:27:36 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1)
130529 13:27:36 [Note] WSREP: RECV thread exiting 0: Success
130529 13:27:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130529 13:27:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:27:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130529 13:27:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:27:36 [Note] WSREP: applier thread exiting (code:0)
130529 13:27:36 [Note] WSREP: recv_thread() joined.
130529 13:27:36 [Note] WSREP: Closing slave action queue.
130529 13:27:36 [Note] WSREP: applier thread exiting (code:5)

And mysqld hangs until killed.

Related branches

Revision history for this message
Jay Janssen (jay-janssen) wrote :

Reproducible in my PXC tutorial environment:

[root@node3 ~]# rpm -qa | grep -i percona
Percona-Server-shared-compat-5.5.30-rel30.2.509.rhel6.i686
percona-toolkit-2.2.2-1.noarch
Percona-XtraDB-Cluster-client-5.5.30-23.7.4.406.rhel6.i686
percona-xtrabackup-2.1.3-608.rhel6.i686
Percona-XtraDB-Cluster-shared-5.5.30-23.7.4.406.rhel6.i686
Percona-XtraDB-Cluster-galera-2.5-1.150.rhel6.i686
Percona-XtraDB-Cluster-server-5.5.30-23.7.4.406.rhel6.i686

And also on a customer server:
root@dev2:~# dpkg -l | grep -i percona
ii libmysqlclient-dev 1:5.5.31-rel30.3-520.precise Percona Server database development files
ii libmysqlclient18 1:5.5.31-rel30.3-520.precise Percona Server database client library
rc percona-server-client-5.5 5.5.28-rel29.2-360.precise Percona Server database client binaries
rc percona-server-server-5.5 5.5.28-rel29.2-360.precise Percona Server database server binaries
ii percona-xtrabackup 2.1.3-608.precise Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.30-23.7.4-405.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.30-23.7.4-405.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 150.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.30-23.7.4-405.precise Percona Server database server binaries

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

As suggested by Seppo, https://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3379 should do. Will apply that and check.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.31-24.8
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
status: New → Triaged
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Fixed. Also, made a few smaller changes.

Changed in percona-xtradb-cluster:
status: Triaged → Fix Committed
Changed in percona-xtradb-cluster:
importance: Undecided → High
Changed in percona-xtradb-cluster:
status: Fix Committed → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-966

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.