Misconfigured HAProxy backend check for aodh

[Daniel Marks]

DISTRIB_ID="OSA"
DISTRIB_RELEASE="15.1.8"
DISTRIB_CODENAME="Ocata"
DISTRIB_DESCRIPTION="OpenStack-Ansible"

When deploying the os_aodh role the backends appear as "DOWN" in the haproxy.

backend aodh_api-back
    mode http
    balance leastconn
    stick store-request src
    stick-table type ip size 256k expire 30m
    option forwardfor
    option httplog
    option httpchk HEAD /
    http-check expect status 401

The HEAD / actually returns "200 OK":

# curl -v http://ctr0002_aodh_container-3e25f851:8042/
* Trying 10.96.224.244...
* Connected to ctr0002_aodh_container-3e25f851 (10.96.224.244) port 8042 (#0)
> GET / HTTP/1.1
> Host: ctr0002_aodh_container-3e25f851:8042
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 27 Sep 2017 12:31:12 GMT
< Server: Apache
< Content-Length: 449
< Content-Type: application/json; charset=UTF-8
<
* Connection #0 to host ctr0002_aodh_container-3e25f851 left intact
{"versions": {"values": [{"status": "stable", "updated": "2013-02-13T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.telemetry-v2+json"}, {"base": "application/xml", "type": "application/vnd.openstack.telemetry-v2+xml"}], "id": "v2", "links": [{"href": "http://ctr0002_aodh_container-3e25f851:8042/v2", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}

The expected status would be returned by a "HEAD /v2" :

# curl --head http://ctr0002_aodh_container-3e25f851:8042/v2
HTTP/1.1 401 Unauthorized
Date: Wed, 27 Sep 2017 12:33:06 GMT
Server: Apache
x-openstack-request-id: req-965115bc-97a9-4d37-967a-47350eb24149
Content-Length: 114
Content-Type: application/json

But as aodh does currently not implement the "/healthcheck" we should stick with "HEAD /" as for most of the other backends.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/507842

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/507843

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507842
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=74acdd5a7bc50de31b86ee10e226c252dd042c6f
Submitter: Jenkins
Branch: master

commit 74acdd5a7bc50de31b86ee10e226c252dd042c6f
Author: Daniel Marks <email address hidden>
Date: Wed Sep 27 14:17:32 2017 +0200

    Fix Aodh's health checks

    HAProxy currently checks aodh backend with "HEAD /" and an expected
    return code of 401. The request actually returns "200 OK". The 401 would
    be returned if the check would target "HEAD /v2", but as all other
    backends that do not implement "/healthcheck" also check "HEAD /" for
    2xx/3xx return codes, I removed the wrongly expected "401".

    Change-Id: If20216c605938c14a94d9b710d3f98b0d885b629
    Closes-Bug: 1719875

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/510551

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507843
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=7954dc4b3f01e0c4a441d1c4186f62993d6b40cc
Submitter: Jenkins
Branch: stable/ocata

commit 7954dc4b3f01e0c4a441d1c4186f62993d6b40cc
Author: Daniel Marks <email address hidden>
Date: Wed Sep 27 14:49:59 2017 +0200

    Fix Aodh's health checks (ocata backport)

    HAProxy currently checks aodh backend with "HEAD /" and an expected
    return code of 401. The request actually returns "200 OK". The 401 would
    be returned if the check would target "HEAD /v2", but as all other
    backends that do not implement "/healthcheck" also check "HEAD /" for
    2xx/3xx return codes, I removed the wrongly expected "401".

    Change-Id: If20216c605938c14a94d9b710d3f98b0d885b629
    Closes-Bug: 1719875

[bhujay kumar (bhatta)]

I faced the same issue in pike and resolved after removing the 'expect 401' . Works fine now . We see the status for this fix is yet to be merged for stable/pike. May be this need to be merged and closed unless this is waiting for some other issues.

[Daniel Marks (d3n14l)]

The gate jobs for stable/pike were broken when I made the commit. In the hope that this is fixed now I triggered a "recheck". See https://review.openstack.org/510551

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 17.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 15.1.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 17.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/510551
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=bfbd7b3233cabc926733829060735cffdc2188c7
Submitter: Zuul
Branch: stable/pike

commit bfbd7b3233cabc926733829060735cffdc2188c7
Author: Daniel Marks <email address hidden>
Date: Mon Oct 9 14:22:54 2017 +0200

    Fix Aodh's health checks (pike backport)

    HAProxy currently checks aodh backend with "HEAD /" and an expected
    return code of 401. The request actually returns "200 OK". The 401 would
    be returned if the check would target "HEAD /v2", but as all other
    backends that do not implement "/healthcheck" also check "HEAD /" for
    2xx/3xx return codes, I removed the wrongly expected "401".

    Change-Id: If20216c605938c14a94d9b710d3f98b0d885b629
    Closes-Bug: 1719875

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 16.0.6 release.