connections to HA-enabled services are not closed properly

[Junien F]

Hi,

Running a xenial/mitaka cloud with 17.11 charms, HA-enabled (with 3 units).

I hooked haproxy logs to a centralized logging system, and thought it'd be nice to have statistics on haproxy termination states. I realized with horror that > 50% of the haproxy connections ended in a "cD" state, which means that the timeout was reached (if you have the same client and server timeout, haproxy will always report a cD state if this timeout was reached, AFAIK).

I investigated more, and found that this is actually benign : the various API calls (from one openstack service to another, say from neutron-api to keystone) are made with "Connection : keep-alive", but the connection is not closed properly by the client, so it stays up for a while until the server closes it. See the wireshark screenshot attached (notice the last packets, sent 1 min and 2 min after the last data byte).

This is causing sockets to be left open needlessly, and also is making haproxy logs useless, which prevents using them in any kind of dashboard.

Thanks

[Junien F (axino)]

Note that this is still happening on xenial/queens with 18.02 charms.