clustercheck hangs when server is hung

[Koa McCullough]

When the mysql server on a node is hung clustercheck will also hang and never fail the server over.

Steps to reproduce:

1) setup a 3 node cluster
2) configure ha_proxy to run with an active writer and 2 backup servers
3) start a sysbench test
4) log into the writer node and run 'kill -SIGSTOP <mysqld_pid>' as root to hang the server

Setup details can be found here: http://www.mysqlperformanceblog.com/2012/06/20/percona-xtradb-cluster-reference-architecture-with-haproxy/

The sysbench test should timeout and the ha_proxy writer vip should also hang.

Attached is a python version of clustercheck that will timeout after 30 seconds and mark the node as failed. I didn't create a branch since this would add a new python dependency to PXC so, I wasn't sure if this needed to be written in perl or not in order to be accepted.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Tested. The existing script can be reused by using timeout in front of mysql like http://sprunge.us/KiRU

Tested the python script too, and it works.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1239