Logger resource temporarily unavailable alert in HAProxy logs

[Éric Lemoine]

There are alert log messages in the HAProxy logs:

    [ALERT] 055/110341 (16) : sendto logger #1 failed: Resource temporarily unavailable (errno=11)

This is related to the log socket not being available when the haproxy process starts. The "wait for log socket" code in docker/base/start.sh is supposed to protect us from that, but it looks like that does not work as expected.

[Éric Lemoine (elemoine)]

Contrary to what's stated in the Bug Description, the problem is not related to the log socket not being available when the haproxy process starts.

The problem is related to net.unix.max_dgram_qlen being too small (10) by default. By increasing net.unix.max_dgram_qlen to 100 (for example) the problem goes away.

See https://bugzilla.redhat.com/show_bug.cgi?id=1257690 for example.

I am going to change the Importance from High to Medium, as the consequence of this issue is just the loss a few log entries at HAProxy startup time.

[Vikram Hosakote (vhosakot)]

This bug may raise false alarm about haproxy to the operator and monitoring tools even though haproxy is running fine and there is no issue.

I will fix this bug by increasing the datagram receive queue length of haproxy's socket.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/292075

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292075
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=9d1911c5a497fe625989f5795676d9f2fa4029e6
Submitter: Jenkins
Branch: master

commit 9d1911c5a497fe625989f5795676d9f2fa4029e6
Author: Vikram Hosakote <email address hidden>
Date: Sun Mar 13 06:47:38 2016 +0000

    Fix false alert messages in HAProxy logs

    Bug #1549753 raises false alarm about haproxy to the operator and
    monitoring tools even though haproxy is running fine and there is
    no issue.

    haproxy uses UDP datagram socket for logging and the default queue
    length of 10 datagrams causes false alert messages in haproxy
    logs. This patch set fixes this bug by increasing the queue length
    to 128.

    Test results of the fix in this patch set are at
    http://paste.openstack.org/show/490302/.

    Change-Id: I64fdd17cd278d51616aa0dfa4e8d2a422c8ee630
    Closes-Bug: #1549753