[RFE] Enable health check httpchk options - haproxy

[Pedro Victor Lourenço Fragola]

In the HA environment keystone uses haproxy but heath check only validates the status of keystone/apache2 TCP ports

backend admin-port_10.5.1.86
    balance leastconn
    server keystone-1 10.5.1.86:35347 check
    server keystone-0 10.5.1.201:35347 check
    server keystone-2 10.5.3.8:35347 check

backend public-port_10.5.1.86
    balance leastconn
    server keystone-1 10.5.1.86:4990 check
    server keystone-0 10.5.1.201:4990 check
    server keystone-2 10.5.3.8:4990 check

This can create a false positive if a unit/apache2 does not have a healthy service and has the TCP port UP, haproxy will continue sending requests to the unit.

It would be a benefit to have the option of a different health check[0] as option httpchk to check a valid apache2 response or to have a custom check using one of the HEAD HTTP, GET or POST HTTP methods.

[0] http://cbonte.github.io/haproxy-dconv/2.0/configuration.html#4-option%20httpchk

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone/+/803502

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-keystone/+/803502
Committed: https://opendev.org/openstack/charm-keystone/commit/579daa68203fe9e5387ca122d2ea72f9c2182ac0
Submitter: "Zuul (22348)"
Branch: master

commit 579daa68203fe9e5387ca122d2ea72f9c2182ac0
Author: John P Lettman <email address hidden>
Date: Wed Aug 4 13:28:21 2021 -0400

    Enable health check httpchk options in haproxy.

    Adds backend options for 'admin-port' and 'public-port' in
    HAProxyContext. HAProxy will now expect 200-300 statuses and the string
    "stable".

    test_haproxy_context_service_enabled updated to reflect expected ctxt.

    Closes-Bug: #1933233
    Change-Id: I88cef4539f5d7dc70f6fbaacfb2ff768e958d346

[Aurelien Lourot (aurelien-lourot)]

Unfortunately we had to revert the previous patch as it was breaking keystone with TLS with:

requests.exceptions.SSLError: HTTPSConnectionPool(host='10.247.39.184', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1129)')))

This can be reproduced by deploying an OpenStack model with Zaza. Zaza will fail this way just after having unsealed the vault, when validating the CA certs. [1]

[1] https://github.com/openstack-charmers/charmed-openstack-tester/blob/master/tests/openstack-upgrade/tests/tests.yaml#L8

[Aurelien Lourot (aurelien-lourot)]

Revert: https://review.opendev.org/c/openstack/charm-keystone/+/804444