Expose prometheus metrics

[Tom Haddon]

Per [1] HAProxy supports exposing a prometheus metrics endpoint. We should look into this and see if it's feasible to do this for the content-cache charm.

[1] https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/

[Tom Haddon (mthaddon)]

It seems like this is only supported in the 2.x series, which is due for release this summer, so this won't be suitable yet.

However, we should see if https://github.com/prometheus/haproxy_exporter is a good option, or whether there's a telegraf plugin for HAProxy that supports similar metrics.

[Tom Haddon (mthaddon)]

We should be able to do this relatively cheaply by looking at the telegraf haproxy relation. This is implemented in the haproxy charm as the statistics relation, the relevant portion of which seems to be https://pastebin.ubuntu.com/p/8qgrkJxKqq/. Obviously there's a little more to it than that as there are functions referenced there, but this should be the approach we take.

[Tom Haddon (mthaddon)]

If we do take this approach, we should be sure to name the relation on the content-cache side in such a way that we could later implement a similar approach for the nginx portion if appropriate, so rather than calling the relation "statistics" as the haproxy charm does, maybe "haproxy-statistics" would be better.

[Nick Moffitt (nick-moffitt)]

We rolled this out and can see `haproxy_rate{juju_env="prod-esm-content-cache"}` metrics in our internal prometheus!