haproxy doesn't care on radosgw crash

[Andrew Lazarev]

Steps to reproduce:
1. Install openstack in HA mode with ceph
2. kill one of rados gateways

Expected behavior:
Nothing happened since we are in HA mode

Observed behavior:
Swift responds with "500 internal error" each third request.
No monitoring system alerted.
haproxy reports that everything is fine.

This happens because rados gateway is installed behind apache and haproxy thinks it is working (apache port is open and responding).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130885

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/130885
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=151ca375000faad75c92a709dee969f0caaeb18b
Submitter: Jenkins
Branch: master

commit 151ca375000faad75c92a709dee969f0caaeb18b
Author: Dmitry Borodaenko <email address hidden>
Date: Fri Oct 24 14:04:43 2014 -0700

    Use haproxy httpchk to detect a radosgw crash

    TCP check by itself is only able to confirm that the Apache process is
    still there and listening on port 6780, an HTTP GET request will get a
    response code 500 when there is no radosgw process behind Apache, and
    haproxy will mark such host as DOWN.

    Change-Id: Iee8e02427d58befa51596867a5c4c623ee0ddbe8
    Closes-Bug: #1385002

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/131927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/131927
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=e2dba5f4d2bfd3489fbb91b4b59662af2f2901fb
Submitter: Jenkins
Branch: stable/5.1

commit e2dba5f4d2bfd3489fbb91b4b59662af2f2901fb
Author: Dmitry Borodaenko <email address hidden>
Date: Fri Oct 24 14:04:43 2014 -0700

    Use haproxy httpchk to detect a radosgw crash

    TCP check by itself is only able to confirm that the Apache process is
    still there and listening on port 6780, an HTTP GET request will get a
    response code 500 when there is no radosgw process behind Apache, and
    haproxy will mark such host as DOWN.

    Change-Id: Iee8e02427d58befa51596867a5c4c623ee0ddbe8
    Closes-Bug: #1385002
    (cherry picked from commit 151ca375000faad75c92a709dee969f0caaeb18b)

[Stanislav Makar (smakar)]

{u'build_id': u'2014-12-09_22-41-06', u'ostf_sha': u'a9afb68710d809570460c29d6c3293219d3624d4', u'build_number': u'49', u'auth_required': True, u'nailgun_sha': u'22bd43b89a17843f9199f92d61fc86cb0f8772f1', u'production': u'docker', u'api': u'1.0', u'fuelmain_sha': u'3aab16667f47dd8384904e27f70f7a87ba15f4ee', u'astute_sha': u'16b252d93be6aaa73030b8100cf8c5ca6a970a91', u'feature_groups': [u'mirantis'], u'release': u'6.0', u'release_versions': {u'2014.2-6.0': {u'VERSION': {u'build_id': u'2014-12-09_22-41-06', u'ostf_sha': u'a9afb68710d809570460c29d6c3293219d3624d4', u'build_number': u'49', u'api': u'1.0', u'nailgun_sha': u'22bd43b89a17843f9199f92d61fc86cb0f8772f1', u'production': u'docker', u'fuelmain_sha': u'3aab16667f47dd8384904e27f70f7a87ba15f4ee', u'astute_sha': u'16b252d93be6aaa73030b8100cf8c5ca6a970a91', u'feature_groups': [u'mirantis'], u'release': u'6.0', u'fuellib_sha': u'2c99931072d951301d395ebd5bf45c8d401301bb'}}}, u'fuellib_sha': u'2c99931072d951301d395ebd5bf45c8d401301bb'}