Percona XtraDB Cluster - HA scalable solution for MySQL

PXC nodes hang on shutdown when using pool-of-threads

Reported by Jay Janssen on 2013-05-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraDB Cluster
Status tracked in Trunk
5.6
Undecided
Unassigned
Trunk
High
Raghavendra D Prabhu

Bug Description

Having thread pool enabled:

thread_handling = pool-of-threads

results in PXC not being able to shutdown gracefully.

Normal shutdown looks like this:
130529 13:26:36 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1)
130529 13:26:36 [Note] WSREP: Received self-leave message.
130529 13:26:36 [Note] WSREP: Flow-control interval: [0, 0]
130529 13:26:36 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130529 13:26:36 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1)
130529 13:26:36 [Note] WSREP: RECV thread exiting 0: Success
130529 13:26:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130529 13:26:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:26:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130529 13:26:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:26:36 [Note] WSREP: applier thread exiting (code:0)
130529 13:26:36 [Note] WSREP: recv_thread() joined.
130529 13:26:36 [Note] WSREP: Closing slave action queue.
130529 13:26:36 [Note] WSREP: applier thread exiting (code:5)
130529 13:26:38 [Note] WSREP: rollbacker thread exiting
130529 13:26:38 [Note] Event Scheduler: Purging the queue. 0 events
130529 13:26:38 [Note] WSREP: dtor state: CLOSED
130529 13:26:38 [Note] WSREP: apply mon: entered 0
130529 13:26:38 [Note] WSREP: apply mon: entered 0
130529 13:26:38 [Note] WSREP: mon: entered 4 oooe fraction 0 oool fraction 0
130529 13:26:38 [Note] WSREP: cert index usage at exit 0
130529 13:26:38 [Note] WSREP: cert trx map usage at exit 0
130529 13:26:38 [Note] WSREP: deps set usage at exit 0
130529 13:26:38 [Note] WSREP: avg deps dist 0
130529 13:26:38 [Note] WSREP: wsdb trx map usage 0 conn query map usage 0
130529 13:26:38 [Note] WSREP: Shifting CLOSED -> DESTROYED (TO: 1)
130529 13:26:38 [Note] WSREP: Flushing memory map to disk...
130529 13:26:38 InnoDB: Starting shutdown...
130529 13:26:42 InnoDB: Shutdown completed; log sequence number 1598630
130529 13:26:42 [Note] /usr/sbin/mysqld: Shutdown complete

130529 13:26:42 mysqld_safe mysqld from pid file /var/lib/mysql/node3.pid ended

Shutdown with thread pool enabled looks like this:

130529 13:27:36 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 1)
130529 13:27:36 [Note] WSREP: Received self-leave message.
130529 13:27:36 [Note] WSREP: Flow-control interval: [0, 0]
130529 13:27:36 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130529 13:27:36 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 1)
130529 13:27:36 [Note] WSREP: RECV thread exiting 0: Success
130529 13:27:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130529 13:27:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:27:36 [Note] WSREP: New cluster view: global state: 9d241cd3-c46c-11e2-0800-81ab1e4b64e6:1, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130529 13:27:36 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130529 13:27:36 [Note] WSREP: applier thread exiting (code:0)
130529 13:27:36 [Note] WSREP: recv_thread() joined.
130529 13:27:36 [Note] WSREP: Closing slave action queue.
130529 13:27:36 [Note] WSREP: applier thread exiting (code:5)

And mysqld hangs until killed.

Jay Janssen (jay-janssen) wrote :

Reproducible in my PXC tutorial environment:

[root@node3 ~]# rpm -qa | grep -i percona
Percona-Server-shared-compat-5.5.30-rel30.2.509.rhel6.i686
percona-toolkit-2.2.2-1.noarch
Percona-XtraDB-Cluster-client-5.5.30-23.7.4.406.rhel6.i686
percona-xtrabackup-2.1.3-608.rhel6.i686
Percona-XtraDB-Cluster-shared-5.5.30-23.7.4.406.rhel6.i686
Percona-XtraDB-Cluster-galera-2.5-1.150.rhel6.i686
Percona-XtraDB-Cluster-server-5.5.30-23.7.4.406.rhel6.i686

And also on a customer server:
root@dev2:~# dpkg -l | grep -i percona
ii libmysqlclient-dev 1:5.5.31-rel30.3-520.precise Percona Server database development files
ii libmysqlclient18 1:5.5.31-rel30.3-520.precise Percona Server database client library
rc percona-server-client-5.5 5.5.28-rel29.2-360.precise Percona Server database client binaries
rc percona-server-server-5.5 5.5.28-rel29.2-360.precise Percona Server database server binaries
ii percona-xtrabackup 2.1.3-608.precise Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.30-23.7.4-405.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.30-23.7.4-405.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 150.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.30-23.7.4-405.precise Percona Server database server binaries

As suggested by Seppo, https://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3379 should do. Will apply that and check.

Changed in percona-xtradb-cluster:
milestone: none → 5.5.31-24.8
assignee: nobody → Raghavendra D Prabhu (raghavendra-prabhu)
status: New → Triaged

Fixed. Also, made a few smaller changes.

Changed in percona-xtradb-cluster:
status: Triaged → Fix Committed
Changed in percona-xtradb-cluster:
importance: Undecided → High
Changed in percona-xtradb-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers