Add health checks for haproxy-configured backends

[Mario Splivalo]

Currently, then charm, then service is deployed, 'hides' it behind the haproxy. If there are multiple units deployed, the haproxy will balance the requests across all the deployed units.

However, only basic backend check is configued. If there is another error with the backend service (say, it returns http/500), haproxy will not consider that as a fault, and the error will be propagated to the calling client.

This can be troubling in multi-unit deployments as the errors might seem intermittent to the calling service (based on how often will haproxy's loadbalancing hit the 'broken' service).

Would it be feasible to implement additional haproxy checks for the configued services? What comes to mind is a plain http-check (for instance: "http-check expect status 200"), or maybe even agent-check for a more sophisticated checks?

Haproxy checks would then use healthcheck from oslo.middleware (https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html) to determine if the configure backend is in good shape to be servicing requests.

Of course, this would hide the issue from the calling service (as haproxy will not loadbalance client requests to the broken backend), so additional nrpe checks should be configured too when service is related with nagios monitoring.

[Peter Sabaini (peter-sabaini)]

Related: Bug #1765739