[RFE] support healthcheck middleware

[Jim Rollenhagen]

Oslo provides a health check middleware that can give details about the server's status: https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html

This is useful for load balancers to determine if a service is up (and add or remove it from rotation), or for monitoring tools to see the health of the server.

Adding this will involve adding the middleware to the wsgi application, and adding the config options into our sample config. We'll also need to
add our own option to enable the middleware, as we don't use paste for enabling and disabling middlewares.

The healthcheck will be unauthenticated, as authenticated healthchecks aren't terribly useful.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/542992

[Jim Rollenhagen (jim-rollenhagen)]

Chatted with nova about this and they tend to think the whole community should decide how we do this.

I was also pointed at the "new" healthcheck middleware spec: https://review.openstack.org/#/c/531456/

So probably some thinking to do here.

[Ruby Loo (rloo)]

17:31 jroll: rloo: I imagine it doesn't need a spec. I also imagine some folks will want to have a larger openstack-wide consensus on it first, based on my conversation with the nova folks this morning. apparently that happened once: https://review.openstack.org/#/c/531456/

17:32 jroll: rloo: I'm just trying to make that middleware available in services that don't use paste, not make a healthcheck that solves all the world's problems :)

[Ruby Loo (rloo)]

Here's my concern. I'm happy to have the coding done to support this middleware. However, the new proposal/spec for a middleware healthcheck will have a different API from the existing one (if I understand it). So what I'd like to know is:

1. is there an easy/simple plan to deprecate the existing/old, and use the new one
2. when do we think, realistically, the new one will be available for use?

I think I could make a better informed decision if I had some idea of the answers for the above two.

So to be clear:
1. this doesn't need a spec
2. I question whether we want to hook in https://docs.openstack.org/oslo.middleware/latest/reference/healthcheck_plugins.html, in light of a new one that has more bells and whistles...

[Ruby Loo (rloo)]

It isn't clear when the new proposal [1] will ever see the light of day (I asked, but didn't get any response. yet.). However, according to them-that-know (see [1]), it should be possible to have both middlewares in ironic. Their APIs are different, so we can support both sets of API at the same time.

Given this information*, I am OK with approving this.

[1] https://review.openstack.org/#/c/531456/

* if the information changes, I reserve the right to change my vote/approval :)

[Julia Kreger (juliaashleykreger)]

Given the tentative language of the spec, and the possible co-existence on the discovery document page defined in it, I suspect these can indeed live together. As such, I'm good with this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542992
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=6f439414bdcef9fc02f844f475ec798d48d42558
Submitter: Zuul
Branch: master

commit 6f439414bdcef9fc02f844f475ec798d48d42558
Author: Jim Rollenhagen <email address hidden>
Date: Fri Feb 9 11:30:24 2018 -0800

    Add optional healthcheck middleware

    This adds the healthcheck middleware from oslo, configurable via the
    [healthcheck]/enabled option. This middleware adds a status check at
    `/healthcheck`.

    This is useful for load balancers to determine if a service is up (and
    add or remove it from rotation), or for monitoring tools to see the
    health of the server.

    This endpoint is unauthenticated, as not all load balancers or
    monitoring tools support authenticating with a health check endpoint.

    Change-Id: I7929d5f4502c3f84b1cf30526c94a458081a6a29
    Closes-Bug: #1748515

[Ruby Loo (rloo)]

NOTE, this is only to add the healthcheck endpoint. We don't want to add a CLI for it [1].

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/%23openstack-ironic.2018-03-19.log.html#t2018-03-19T14:26:29

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 11.0.0 release.