Neutron haproxy logs are not being collected

[Daniel Alvarez]

In Neutron, we use haproxy to proxy metadata requests from instances to Nova Metadata service.
By default, haproxy logs to /dev/log but in Ubuntu, those requests get redirected by rsyslog to
/var/log/haproxy.log which is not being collected.

ubuntu@devstack:~$ cat /etc/rsyslog.d/49-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
if $programname startswith 'haproxy' then /var/log/haproxy.log
&~

Another possibility would be to change the haproxy.cfg file to include the log-tag option so that haproxy uses a different tag [0] and then it'd be dumped into syslog instead but this would break backwards compatibility.

[0] https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#3.1-log-tag

[Brian Haley (brian-haley)]

So I tested using log-tag in the haproxy conf file, and it does help differentiating things in /var/log/haproxy.log since all the haproxies will log there.

A typical line went from this:

Jan 15 16:57:45 localhost haproxy[14095]: 10.0.0.6:34898 [15/Jan/2018:16:57:45.650] listener listener/metadata 2/0/1/17/20 200 157 - - ---- 1/1/0/1/0 0/0 "GET /2009-04-04/meta-data/hostname HTTP/1.1"

to this:

Jan 19 14:49:34 localhost haproxy-metadata-proxy-c891c6a2-07b3-4cdf-86c7-6164cdac7dd1[7418]: 10.0.0.7:52947 [19/Jan/2018:14:49:34.097] listener listener/metadata 7/0/0/368/375 200 157 - - ---- 1/1/0/0/0 0/0 "GET /2009-04-04/meta-data/hostname HTTP/1.1"

Which doesn't look too backwards incompatible, just a change in what proxy logged it.

[Brian Haley (brian-haley)]

So I think these are logged, they're just not where we expect them to be. I noticed that when running journalctl in devstack:

    'sudo journalctl --unit <email address hidden>'

I was seeing haproxy lines, most likely since it's using systemd and haproxy is started as a child.

I also noticed it in other jobs as well:

http://logs.openstack.org/88/535488/1/check/barbican-simple-crypto-devstack-tempest/4a4b91a/logs/screen-q-l3.txt.gz#_Jan_18_22_10_24_019350

That doesn't mean there isn't work to do in TripleO to capture them as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/535948

[Ihar Hrachyshka (ihar-hrachyshka)]

I guess having haproxy prefix is backwards compatible enough to allow it to slip like that.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535948
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=703ff85b8262997f209e7666396c5d430d3baa34
Submitter: Zuul
Branch: master

commit 703ff85b8262997f209e7666396c5d430d3baa34
Author: Brian Haley <email address hidden>
Date: Fri Jan 19 15:53:38 2018 -0500

    Add log-tag to haproxy config file

    By adding a log-tag line to the haproxy config file that contains
    the network or router id, we will be able to differentiate which
    proxy is logging what. This should help with debugging.

    Change-Id: I5bb57b7682c00645e20cce69847dbb3b72165aa8
    Partial-bug: #1744359

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/536919

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536919
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f74b39c57dbb306dd3bb9a68cc44491d54b7988a
Submitter: Zuul
Branch: stable/pike

commit f74b39c57dbb306dd3bb9a68cc44491d54b7988a
Author: Brian Haley <email address hidden>
Date: Fri Jan 19 15:53:38 2018 -0500

    Add log-tag to haproxy config file

    By adding a log-tag line to the haproxy config file that contains
    the network or router id, we will be able to differentiate which
    proxy is logging what. This should help with debugging.

    Change-Id: I5bb57b7682c00645e20cce69847dbb3b72165aa8
    Partial-bug: #1744359
    (cherry picked from commit 703ff85b8262997f209e7666396c5d430d3baa34)

[Akihiro Motoki (amotoki)]

Brian, is there anything remaining in neutron? You used "Partial-Bug" for a neutron fix.

[Brian Haley (brian-haley)]

Akihiro - yes, I don't think there is more in neutron, I thought there was at the time or I would have used closes-bug instead. We should now have slightly better logging in this area.

[Dr. Jens Harbott (j-harbott)]

Seems like this has been an issue in Neutron, please reopen if there is still some issue with devstack.