layer 7 haproxy backend checks

[Andrea Ieri]

The current haproxy config adds health checks for all backends. These are however the default TCP checks[0], which succeed simply if a connection to the specified port is possible, and don't go any further in trying to assess if the backend service is actually healthy.

In some partial failure scenarios - e.g. extreme load or memory pressure - TCP checks may succeed where more thorough ones would not, causing API traffic to be routed to backends that aren't actually able to process it.

HAproxy offers the possibility of reconfiguring the layer 4 checks into layer 7 ones by using the httpchk option[1]. This is still fairly basic, but configuring the http-check option[4] makes the check a bit more accurate, while service-specific checks are available for some backends such as mysql[2] and postgres[3]. Finally, external scripts[5] can be invoked for a fully customized check routine.

In order to improve the control plane resilience we should gradually switch to more advanced backend checks. Plain TCP checks should remain the default, but principal charms should provide more accurate check specifications via their relation to hacluster. Depending on the service, a properly configured httpchk may suffice, while in some cases a fully custom script may be more appropriate.

This is not in contrast with giving more control to pacemaker via openstack resource agents, as described in bug 1880611.

[0] https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#4-option%20httpchk
[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20mysql-check
[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20pgsql-check
[4] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-check%20expect
[5] https://www.haproxy.com/documentation/hapee/latest/onepage/#external-check%20command

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/750224

[Jose Guedez (jfguedez)]

Marking invalid for 'hacluster' as it doesn't handle haproxy configurations (this is handled by the principal charms)

[Jose Guedez (jfguedez)]

commenting to send notification to field:medium

[OpenStack Infra (hudson-openstack)]

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/750224
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Andrea Ieri (aieri)]

for the record this bug affects all principals using hacluster, so the bug may be in a shared library

[Jose Guedez (jfguedez)]

We had another event when having the L7 checks would have been helpful. Since the balancing scheme is "leastconn" [0], having a bad backend that is not completely dead (i.e. accepts network connections, but just never replies until a timeout is reached), will lead to the situation of new connections to the loadbalancer being pointed to a "broken" backend.

We have recently seen this in production clouds using the gnocchi charm, but should be similar for other services (principal charms). In this particular case the requests were being continuously pointed to the unresponsive backend, even though the others were healthy (just 1 out of 3 backends was unhealthy). Since the unhealthy gnocchi backend service was still accepting network connections, haproxy continued sending it requests. Pausing the bad backend restored service to clients while the problem with the unhealthy backend was resolved.

Having a L7 check would make haproxy realise that a backend is unhealthy quickly and requests would be then be routed to the other 2 healthy backends. This would be similar to how readiness checks in Kubernetes (typically L7), would prevent this situation from happening.

[0] https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/templates/haproxy.cfg#L79

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/862521

[Gabriel Cocenza (gabrielcocenza)]

I've opened two PR against charm-layer-openstack-api[0] and charm-layer-openstack[1] in order to give layer 7 haproxy backend checks on reactive openstack-api charms

[0] https://review.opendev.org/c/openstack/charm-layer-openstack-api/+/862750
[1] https://review.opendev.org/c/openstack/charm-layer-openstack/+/862752

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/862521
Committed: https://opendev.org/openstack/charm-glance/commit/96a2bb3af9a9b3b5756e5d618371187fa6e09b8f
Submitter: "Zuul (22348)"
Branch: master

commit 96a2bb3af9a9b3b5756e5d618371187fa6e09b8f
Author: Gabriel Cocenza <email address hidden>
Date: Mon Oct 24 12:04:05 2022 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: I81d7fd67029dd5025f95b41d788b03ce4b6038bb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-gnocchi/+/874030

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone/+/874035

[Gabriel Cocenza (gabrielcocenza)]

charm-ceilometer removed haproxy since ocata, so it doesn't make sense to add layer 7 check to this charm.

[Gabriel Cocenza (gabrielcocenza)]

charm-openstack-dashboard has its own haproxy.cfg template that it's not from charmhelpers.
The template uses `listen` for dash_insecure and dash_secure and as the documentation says[0]:

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.

[0] http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874177

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-designate/+/874247

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-heat/+/874262

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/874967

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-swift-proxy/+/874973

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/875299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/874967
Committed: https://opendev.org/openstack/charm-ironic-api/commit/65379959f2bcb9d317f1ece0b07ab887bf8b47c9
Submitter: "Zuul (22348)"
Branch: master

commit 65379959f2bcb9d317f1ece0b07ab887bf8b47c9
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 15:02:32 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: I5544d9b4c5b9030c0e5b619829152edd01fa0889

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-keystone/+/874035
Committed: https://opendev.org/openstack/charm-keystone/commit/04480c4ff4b37ad6aef619d452522721a728b587
Submitter: "Zuul (22348)"
Branch: master

commit 04480c4ff4b37ad6aef619d452522721a728b587
Author: Gabriel Cocenza <email address hidden>
Date: Wed Feb 15 19:09:25 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: I50a9442ae66da71793a5e9904d23c26d1fbbdf42

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-designate/+/874247
Committed: https://opendev.org/openstack/charm-designate/commit/6711b12054e2da999bf7f606344f6bb9af2768b7
Submitter: "Zuul (22348)"
Branch: master

commit 6711b12054e2da999bf7f606344f6bb9af2768b7
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 10:21:23 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default).

    Closes-Bug: #1880610
    Change-Id: Ibadd5648cdf69eef42b05e29772dcae9897d668b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-swift-proxy/+/874973
Committed: https://opendev.org/openstack/charm-swift-proxy/commit/c29a1c7b8e953b2055763c16d04786b800300f9d
Submitter: "Zuul (22348)"
Branch: master

commit c29a1c7b8e953b2055763c16d04786b800300f9d
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 23 21:06:33 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: Ie1f53d38c1c45ce3b2ddf5041eaa04a7f3139066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-heat/+/874262
Committed: https://opendev.org/openstack/charm-heat/commit/19cf71dc79bcba91132dcd769eeb99e18b09b754
Submitter: "Zuul (22348)"
Branch: master

commit 19cf71dc79bcba91132dcd769eeb99e18b09b754
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 11:38:13 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for heat-api and heat-cfn-api on openstack releases >= queens.

    Closes-Bug: #1880610
    Change-Id: I94c9418c82cdddd5a5d9ed400ab47889bfb225b1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-gnocchi/+/874030
Committed: https://opendev.org/openstack/charm-gnocchi/commit/b33b6a213f6768072d21827ea38b6d28f35f9d0e
Submitter: "Zuul (22348)"
Branch: master

commit b33b6a213f6768072d21827ea38b6d28f35f9d0e
Author: Gabriel Cocenza <email address hidden>
Date: Wed Feb 15 17:54:19 2023 -0300

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: If7841f710d0dfffdc7ee6b424cb77fc34aeae823

[Felipe Reyes (freyes)]

Hi Gabriel, thanks for the large piece of work you've done to support healthchecks, since we wouldn't want this new functionality goes inadvertently released, I added a task for the charm-guide, so this get documented in the release-notes ;-)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api/+/874309
Committed: https://opendev.org/openstack/charm-neutron-api/commit/7330c5badfa72ac2112389838f6e4b64d3f45e3c
Submitter: "Zuul (22348)"
Branch: master

commit 7330c5badfa72ac2112389838f6e4b64d3f45e3c
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 15:27:55 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for neutron-api.

    Closes-Bug: #1880610

    Change-Id: Ia820d8c2ca709d6b358b1c80d770624568d9a85b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/875299
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/468d648655f420230f3018123ac0ec29bdb0ca87
Submitter: "Zuul (22348)"
Branch: master

commit 468d648655f420230f3018123ac0ec29bdb0ca87
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 23 21:17:56 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: I4a947c5b52eb3283c08a0d39cc9bf14695a63eab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/874177
Committed: https://opendev.org/openstack/charm-cinder/commit/51e32a4f7e8d7256fd6189197b49f1965f63199b
Submitter: "Zuul (22348)"
Branch: master

commit 51e32a4f7e8d7256fd6189197b49f1965f63199b
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 16 18:18:29 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for cinder-api on openstack releases >= ocata.

    Closes-Bug: #1880610
    Change-Id: I9d118f70fc1390be7b800ad20ae20e77818adac7

[Peter Matulis (petermatulis)]

Hi Alex, it would be very helpful if you could add a summary on how this affects the Charm Guide.

[Alex Kavanagh (ajkavanagh)]

@petermatulis essentially that the checks performed when in HA for the charms linked in this bug do layer 7 http checks (looking for a 200) rather than just a layer 4 (?) tcp port open check. i.e. that there is at least something listening on the port that can respond with a http OK.

However, this doesn't guarantee that the service is actually working as the service may then immediately error due to, say, database issues. i.e. the checks are a guide, and better than they were, but aren't the complete story.