Large number of TIME_WAIT connections from haproxy cause stalled API service requests

[Dmitry Borodaenko]

At load, haproxy begins accumulating a large number of connections to backend services stuck in TIME_WAIT state. Eventually, this leads to stalled requests to OpenStack API services.

Setting the http-server-close option:
https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20http-server-close

makes the problem disappear.

[Dmitriy Novakovskiy (dnovakovskiy)]

 Should the http-server-close option be propagated to customers experiencing the issue? I know 1 for sure

[Bogdan Dobrelya (bogdando)]

The http-server-close is enabled by default according to the docs. Are you sure enabling it explicitly would help?

[Bogdan Dobrelya (bogdando)]

My mistake, this option looks like disabled by defaults and we may want it enabled both in a frontend and in a backend indeed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149963
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6c6bfc35f779774141274579b948280518f57e2d
Submitter: Jenkins
Branch: master

commit 6c6bfc35f779774141274579b948280518f57e2d
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Jan 26 11:06:08 2015 +0100

    Use haproxy's http-server-close by default

    At load, haproxy begins accumulating a large number of connections to
    backend services stuck in TIME_WAIT state. Eventually, this leads to
    stalled requests to OpenStack API services. http-server-close solves
    that problem.

    Change-Id: Ie405fe2d384d9d8375f60ad2c3c9885f41fe6d49
    Closes-Bug: 1413104

[Mike Scherbakov (mihgen)]

Don't we want to backport this to older branches?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/152059

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/152060

[Bartłomiej Piotrowski (bpiotrowski)]

We do, I completely forgot about it.

[Bartłomiej Piotrowski (bpiotrowski)]

6.0: https://review.openstack.org/#/c/152059/
5.1: https://review.openstack.org/#/c/152060/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152059
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f41f594850fb8ced89a2a29649cd478492d2b29b
Submitter: Jenkins
Branch: stable/6.0

commit f41f594850fb8ced89a2a29649cd478492d2b29b
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Jan 26 11:06:08 2015 +0100

    Use haproxy's http-server-close by default

    At load, haproxy begins accumulating a large number of connections to
    backend services stuck in TIME_WAIT state. Eventually, this leads to
    stalled requests to OpenStack API services. http-server-close solves
    that problem.

    Change-Id: Ie405fe2d384d9d8375f60ad2c3c9885f41fe6d49
    Closes-Bug: 1413104
    (cherry picked from commit 6c6bfc35f779774141274579b948280518f57e2d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/152060
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a8417ef811919dbe7abee90eeb67f4ca7ede2935
Submitter: Jenkins
Branch: stable/5.1

commit a8417ef811919dbe7abee90eeb67f4ca7ede2935
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Jan 26 11:06:08 2015 +0100

    Use haproxy's http-server-close by default

    At load, haproxy begins accumulating a large number of connections to
    backend services stuck in TIME_WAIT state. Eventually, this leads to
    stalled requests to OpenStack API services. http-server-close solves
    that problem.

    Change-Id: Ie405fe2d384d9d8375f60ad2c3c9885f41fe6d49
    Closes-Bug: 1413104
    (cherry picked from commit 6c6bfc35f779774141274579b948280518f57e2d)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/156483

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156483
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cbaae5b5fd0a6c7edc562f6dcc9d48ca568bb600
Submitter: Jenkins
Branch: stable/5.0

commit cbaae5b5fd0a6c7edc562f6dcc9d48ca568bb600
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Jan 26 11:06:08 2015 +0100

    Use haproxy's http-server-close by default

    At load, haproxy begins accumulating a large number of connections to
    backend services stuck in TIME_WAIT state. Eventually, this leads to
    stalled requests to OpenStack API services. http-server-close solves
    that problem.

    Change-Id: Ie405fe2d384d9d8375f60ad2c3c9885f41fe6d49
    Closes-Bug: 1413104
    (cherry picked from commit 6c6bfc35f779774141274579b948280518f57e2d)
    (cherry picked from commit a8417ef811919dbe7abee90eeb67f4ca7ede2935)

[Dina Belova (dbelova)]

Still needs to be verified on the scale lab.

[Sergey Novikov (snovikov)]

Verified on fuel-6.1-445-2015-05-20_22-10-04.iso.

Steps to verify:
    1. Deploy cluster (HA, Neutron+VLAN, 3 controllers + 2 compute)
    2. Run OSTF tests
    3. Check haproxy config file - option 'http-server-close' is setted
    4. Simulate load at haproxy (a large number of request to some Openstack service in multiple threads)
    5. Check haproxy status (status is shown on <public_vip_ip>:10000 by default)