Haproxy doesn't react to communications problems with mysql

[Dennis Dmitriev]

If a controller is available thru the management network but ports 3307 or 4567 are inaccessible for some reason (routing problem, iptables or network settings unable packets to these ports) than Haproxy unable to detect that the mysql backend doesn't work.

This cause failure of most of cluster operations if this is the primary controller where mysql service was affected.

Steps to reproduce:
    - Deploy HA cluster;
    - SSH to primary controller, block access to port 3307 and/or 4567 via iptables;

Expected results:
    - pacemaker mark resource [p_mysql] as 'Stopped' for primary controller;
    - haproxy mark 'mysqld' backend as 'DOWN' for primary controller

Actual result:
    - pacemaker shows the [p_mysql] resource as 'Started' on primary controller
    - haproxy tries deliver requests to the inaccessible 'mysqld' backend

This caused because the OCF script that pacemaker uses (/usr/lib/ocf/resource.d/mirantis/mysql-wss) and the script that haproxy uses thru xinetd (/usr/local/bin/clustercheck) both check the mysql only via local socket file. It is never checked for necessary ports are open.

[Sergii Golovatiuk (sgolovatiuk)]

I am marking this as invalid, as HAProxy uses different technique for detecting issues with MySQL

HaProxy uses galeracheck script via port 49000. Configuration can be found at /etc/haproxy/conf.d/110-mysqld.cfg

galeracheck is a simple bash script running via xinetd which generates the response in HTTP format.

xinetd configuration can be found at /etc/xinetd.d/galeracheck. galeracheck itself can be found at /usr/local/bin/clustercheck

galeracheck uses unix socket as it's default configuration for mysql client. If to block port 3307 it won't detect it ad it requires some modification to set protocol=tcp for mysql client. If to block port 4567 only, after 1-2 minutes this node will be marked as down.