Startup script on Debian shows "Failed" too soon

[Miguel Angel Nieto]

When I start a node it needs to get a IST or SST from other nodes. This is usually a long process, but the script shows "Failed" without waiting to the process to finish:

root@PXC3:~# /etc/init.d/mysql start
Starting MySQL (Percona XtraDB Cluster) database server: mysqld . . . . . . . . . . . . . . failed!
root@PXC3:~# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
root@PXC3:~#

...wait... some time

root@PXC3:~# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 5.5.28-55 Percona XtraDB Cluster (GPL), wsrep_23.7.r3821
[...]

[Raghavendra D Prabhu (raghavendra-prabhu)]

It is not possible to accurately determine when the SST will be
complete since it depends on data size etc.

So, the solution here is to block the COM_PING till the server is
fully up, the blocking can be at unix socket level.

This situation is similar to how innodb blocking buffer pool
restore works.

[MentalPower (mentalpower)]

We do the following in our bootstrap and initialization scripts, in case anyone can use it:

/etc/init.d/mysql start
while [[ "$(pidof mysqld)" && `clustercheck | grep 503` ]]; do
    echo "Waiting for cluster to synchronize...";
    sleep 5;
done

It obviously requires a properly configured clustercheck user.

GRANT PROCESS on *.* TO 'clustercheckuser'@'localhost' IDENTIFIED BY 'somePassword';

[Raghavendra D Prabhu (raghavendra-prabhu)]

After discussion we decided to:

a) Have a --init-pid which passes the init pid from init-script
to mysqld.

b) Whenever there is a blocking operation, mysqld sends a signal
to this script which is trapped by it. Another such signal is
sent only when first is sent.
The timeout will be adjusted accordingly based on this.

[dannie (daniz)]

Looking forward to test and confirm from my end.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1279