haproxy_server: rsyslog unable to log haproxy locally

[Corey Wright]

HAProxy logs are not written locally by rsyslog as configured by the haproxy_server role, at least on Ubuntu 16.04/"Xenial", because:
1. Rsyslog runs as user/group unable to write to OSA's /var/log/haproxy symlink target directory (eg /openstack/log/infra1-haproxy).
2. HAProxy package's rsyslog config takes priority over OSA's HAProxy rsyslog config (ie 49-haproxy.conf vs 99-haproxy-local-logging.conf).

This problem is applicable to stable/pike, stable/queens, and master branches of openstack-ansible-haproxy_server.

1. Rsyslog vs /var/log/haproxy

Rsyslog can't write to /var/log/haproxy (technically the target directory of that symlink, but as that directory varies and is based on whether HAProxy is running on "infra" nodes or a dedicated "loadbalancer" node, eg

root@infra1:~# ps -o pid,user,group,args -p $(systemctl status rsyslog.service | awk '/Main PID:/ { print $3; }')
  PID USER GROUP COMMAND
15948 syslog syslog /usr/sbin/rsyslogd -n

root@infra1:~# ls -ld /var/log/haproxy $(realpath /var/log/haproxy)
drwxr-xr-x 2 haproxy adm 4096 Jul 26 05:16 /openstack/log/infra1-haproxy
lrwxrwxrwx 1 haproxy adm 29 Jul 26 05:16 /var/log/haproxy -> /openstack/log/infra1-haproxy

So while rsyslog runs as "syslog:syslog" (in chown syntax), the directory it is suppose to write to is "haproxy:adm" as set by the haproxy_server role, which is incompatible.

2. OSA vs Ubuntu/Debian package HAProxy Rsyslog config

The HAProxy package installs /etc/rsyslog.d/49-haproxy.conf which ends with "&~" and stops processing of log submissions from haproxy before getting to OSA's logging directives in /etc/rsyslog.d/99-haproxy-local-logging.conf (due to ordering of config files by file name).

root@infra1:~# ls -l /etc/rsyslog.d/*haproxy*.conf
-rw-r--r-- 1 root root 282 Jun 14 2016 /etc/rsyslog.d/49-haproxy.conf
-rw-r--r-- 1 root root 171 Jul 26 05:16 /etc/rsyslog.d/99-haproxy-local-logging.conf

root@infra1:~# dpkg -S /etc/rsyslog.d/49-haproxy.conf
haproxy: /etc/rsyslog.d/49-haproxy.conf

root@infra1:~# apt-cache policy haproxy
haproxy:
  Installed: 1.6.3-1ubuntu0.1
  Candidate: 1.6.3-1ubuntu0.1
  Version table:
 *** 1.6.3-1ubuntu0.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.6.3-1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

root@infra1:~# tail -n3 /etc/rsyslog.d/49-haproxy.conf
# Send HAProxy messages to a dedicated logfile
if $programname startswith 'haproxy' then /var/log/haproxy.log
&~

[Corey Wright (coreywright)]

argh, too many interruptions while writing/editing the bug report resulted in an incomplete paragraph:

Rsyslog can't write to /var/log/haproxy (technically the target directory of that symlink, but as that directory varies and is based on whether HAProxy is running on "infra" nodes or a dedicated "loadbalancer" node, eg...

/openstack/log/${hostname}-haproxy, so I'll simply use the symlink name, however technically incorrect).

[Corey Wright (coreywright)]

testing methodology of (and within) openstack-ansible-haproxy_server Git repo

1. create Ansible "roles" directory symlinked back to haproxy_server Git repo

mkdir roles
ln -s ../ roles/haproxy_server

2. create haproxy_server role deployment playbook

cat <<EOF >test.yml
---
- name: Playbook for role testing
  hosts: all
  user: root
  roles:
    - role: "haproxy_server"
  vars:
    - haproxy_hatop_downloader: '{{ inventory_hostname }}'
    - external_lb_vip_address: '127.0.0.1'
EOF

3. create inventory

cat <<EOF >hosts
# centos-7
a.b.c.d
# opensuse-42.3
e.f.g.h
# ubuntu-16.04
i.j.k.l
# ubuntu-18.04
m.n.o.p
EOF

4. create ansible config

cat <<EOF >ansible.cfg
[defaults]
hostfile = hosts
EOF

5. deploy haproxy_server role

ansible-playbook test.yml -l a.b.c.d

6. inspect haproxy_server role deploy

ssh root@a.b.c.d

[Corey Wright (coreywright)]

Ubuntu 16.04 with haproxy_server master

user@host:~/openstack-ansible-haproxy_server$ git describe # ie master
18.0.0.0b1-16-g7015fc3

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.84

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.84

root@ubuntu-1604-haproxy-server:~# (source /etc/os-release; echo ${PRETTY_NAME})
Ubuntu 16.04.3 LTS

root@ubuntu-1604-haproxy-server:~# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
13832 syslog syslog unconfined /usr/sbin/rsyslogd -n

root@ubuntu-1604-haproxy-server:~# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:29 /var/log/haproxy
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:29 /var/log/haproxy/.

root@ubuntu-1604-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1604-haproxy-server:~# date --rfc-3339=seconds
2018-07-31 18:37:55+00:00

root@ubuntu-1604-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
root@ubuntu-1604-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

root@ubuntu-1604-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1604-haproxy-server:~# grep -il haproxy /etc/rsyslog.conf /etc/rsyslog.d/*.conf
/etc/rsyslog.d/49-haproxy.conf
/etc/rsyslog.d/99-haproxy-local-logging.conf

root@ubuntu-1604-haproxy-server:~# cat /etc/rsyslog.d/49-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
if $programname startswith 'haproxy' then /var/log/haproxy.log
&~

root@ubuntu-1604-haproxy-server:~# tail -n2 /var/log/haproxy.log
Jul 31 18:38:18 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log
Jul 31 18:38:25 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

root@ubuntu-1604-haproxy-server:~# # it's getting logged, but not to our log files
root@ubuntu-1604-haproxy-server:~# # let's demonstrate how to fix this

root@ub...

[Corey Wright (coreywright)]

user@host:~/openstack-ansible-haproxy_server$ git describe # ie master
18.0.0.0b1-16-g7015fc3

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.126

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.126

root@ubuntu-1804-haproxy-server:~# (source /etc/os-release; echo ${PRETTY_NAME})
Ubuntu 18.04 LTS

root@ubuntu-1804-haproxy-server:~# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
15076 syslog syslog unconfined /usr/sbin/rsyslogd -n

root@ubuntu-1804-haproxy-server:~# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:51 /var/log/haproxy
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:51 /var/log/haproxy/.

root@ubuntu-1804-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1804-haproxy-server:~# date --rfc-3339=seconds
2018-07-31 18:55:57+00:00

root@ubuntu-1804-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
root@ubuntu-1804-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

root@ubuntu-1804-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1804-haproxy-server:~# grep -il haproxy /etc/rsyslog.conf /etc/rsyslog.d/*.conf
/etc/rsyslog.d/49-haproxy.conf
/etc/rsyslog.d/99-haproxy-local-logging.conf

root@ubuntu-1804-haproxy-server:~# cat /etc/rsyslog.d/49-haproxy.conf
# Create an additional socket in haproxy's chroot in order to allow logging via
# /dev/log to chroot'ed HAProxy processes
$AddUnixListenSocket /var/lib/haproxy/dev/log

# Send HAProxy messages to a dedicated logfile
if $programname startswith 'haproxy' then /var/log/haproxy.log
&~

root@ubuntu-1804-haproxy-server:~# tail -n2 /var/log/haproxy.log
Jul 31 18:56:09 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log
Jul 31 18:56:16 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

root@ubuntu-1604-haproxy-server:~# # same problem and fix as ubuntu 16.04

root@ubuntu-1804-haproxy-server:~# mv /etc/rsyslog.d/{99,10}-haproxy-local-logging.conf

root@ubuntu-1804-haproxy-server:~# syst...

[Corey Wright (coreywright)]

CentOS 7 with git master: No bug.

user@host:~/openstack-ansible-haproxy_server$ git describe # ie master
18.0.0.0b1-16-g7015fc3

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.125

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.125

[root@centos-7-haproxy-server ~]# (source /etc/os-release; echo ${PRETTY_NAME})
CentOS Linux 7 (Core)

[root@centos-7-haproxy-server ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

[root@centos-7-haproxy-server ~]# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
17562 root root system_u:system_r:syslogd_t:s0 /usr/sbin/rsyslogd -n

[root@centos-7-haproxy-server ~]# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x. haproxy adm unconfined_u:object_r:var_log_t:s0 /var/log/haproxy
drwxr-xr-x. haproxy adm unconfined_u:object_r:var_log_t:s0 /var/log/haproxy/.

[root@centos-7-haproxy-server ~]# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open ‘/var/log/haproxy/haproxy.log’ for reading: No such file or directory
tail: cannot open ‘/var/log/haproxy/haproxy-status.log’ for reading: No such file or directory

[root@centos-7-haproxy-server ~]# date --rfc-3339=seconds
2018-07-31 17:48:50+00:00

[root@centos-7-haproxy-server ~]# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
[root@centos-7-haproxy-server ~]# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

[root@centos-7-haproxy-server ~]# tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
Jul 31 17:49:20 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
Jul 31 17:49:24 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[Corey Wright (coreywright)]

openSUSE 42.3 with git master: No bug.

user@host:~/openstack-ansible-haproxy_server$$ git describe # ie master
18.0.0.0b1-16-g7015fc3

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1

user@host:~/openstack-ansible-haproxy_server$$ ansible-playbook test.yml -l 172.20.41.106

user@host:~/openstack-ansible-haproxy_server$$ ssh root@172.20.41.106

opensuse-423-haproxy-server:~ # (source /etc/os-release; echo ${PRETTY_NAME})
openSUSE Leap 42.3

opensuse-423-haproxy-server:~ # ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
23079 root root unconfined /usr/sbin/rsyslogd -n

opensuse-423-haproxy-server:~ # ls -ldZ /var/log/haproxy
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:11 /var/log/haproxy
drwxr-xr-x 2 haproxy adm ? 4096 Jul 31 18:11 /var/log/haproxy/.

opensuse-423-haproxy-server:~ # tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

opensuse-423-haproxy-server:~ # date --rfc-3339=seconds
2018-07-31 18:17:49+00:00

opensuse-423-haproxy-server:~ # python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
opensuse-423-haproxy-server:~ # python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

opensuse-423-haproxy-server:~ # tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
2018-07-31T18:18:12.714195+00:00 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
2018-07-31T18:18:19.882034+00:00 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[Corey Wright (coreywright)]

tl;dr haproxy logging by way of rsyslog works on ubuntu 16.04 after applying proposed commit.

user@host:~/openstack-ansible-haproxy_server$ git describe # ie bug/1783886
18.0.0.0b1-17-ga412e24

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ # yes, deleted & (re)created vm and got same ip as before
user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.84

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.84

root@ubuntu-1604-haproxy-server:~# (source /etc/os-release; echo ${PRETTY_NAME})
Ubuntu 16.04.3 LTS

root@ubuntu-1604-haproxy-server:~# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
 4241 syslog syslog unconfined /usr/sbin/rsyslogd -n

root@ubuntu-1604-haproxy-server:~# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x 2 syslog adm ? 4096 Aug 1 04:04 /var/log/haproxy
drwxr-xr-x 2 syslog adm ? 4096 Aug 1 04:04 /var/log/haproxy/.

root@ubuntu-1604-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1604-haproxy-server:~# date --rfc-3339=seconds
2018-08-01 04:15:33+00:00

root@ubuntu-1604-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
root@ubuntu-1604-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

root@ubuntu-1604-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
Aug 1 04:15:46 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
Aug 1 04:15:54 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[Corey Wright (coreywright)]

tl;dr haproxy logging by way of rsyslog works on ubuntu 18.04 after applying proposed commit.

user@host:~/openstack-ansible-haproxy_server$ git describe # ie bug/1783886
18.0.0.0b1-17-ga412e24

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.103

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.103

root@ubuntu-1804-haproxy-server:~# (source /etc/os-release; echo ${PRETTY_NAME})
Ubuntu 18.04 LTS

root@ubuntu-1804-haproxy-server:~# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
 3725 syslog syslog unconfined /usr/sbin/rsyslogd -n

root@ubuntu-1804-haproxy-server:~# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x 2 syslog adm ? 4096 Aug 1 04:24 /var/log/haproxy
drwxr-xr-x 2 syslog adm ? 4096 Aug 1 04:24 /var/log/haproxy/.

root@ubuntu-1804-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

root@ubuntu-1804-haproxy-server:~# date --rfc-3339=seconds
2018-08-01 04:29:33+00:00

root@ubuntu-1804-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
root@ubuntu-1804-haproxy-server:~# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

root@ubuntu-1804-haproxy-server:~# tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
Aug 1 04:29:42 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
Aug 1 04:29:49 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[Corey Wright (coreywright)]

tl;dr haproxy logging by way of rsyslog continues to work on centos 7 after applying proposed commit.

user@host:~/openstack-ansible-haproxy_server$ git describe # ie bug/1783886
18.0.0.0b1-17-ga412e24

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.130

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.130

[root@centos-7-haproxy-server ~]# (source /etc/os-release; echo ${PRETTY_NAME})
CentOS Linux 7 (Core)

[root@centos-7-haproxy-server ~]# ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
10460 root root system_u:system_r:syslogd_t:s0 /usr/sbin/rsyslogd -n

[root@centos-7-haproxy-server ~]# ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x. haproxy adm unconfined_u:object_r:var_log_t:s0 /var/log/haproxy
drwxr-xr-x. haproxy adm unconfined_u:object_r:var_log_t:s0 /var/log/haproxy/.

[root@centos-7-haproxy-server ~]# tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open ‘/var/log/haproxy/haproxy.log’ for reading: No such file or directory
tail: cannot open ‘/var/log/haproxy/haproxy-status.log’ for reading: No such file or directory

[root@centos-7-haproxy-server ~]# date --rfc-3339=seconds
2018-08-01 04:43:12+00:00

[root@centos-7-haproxy-server ~]# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
[root@centos-7-haproxy-server ~]# python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

[root@centos-7-haproxy-server ~]# tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
Aug 1 04:43:24 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
Aug 1 04:43:30 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[Corey Wright (coreywright)]

tl;dr haproxy logging by way of rsyslog continues to work on opensuse 42.3 after applying proposed commit.

user@host:~/openstack-ansible-haproxy_server$ git describe # ie bug/1783886
18.0.0.0b1-17-ga412e24

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook --version | head -n1
ansible-playbook 2.6.2

user@host:~/openstack-ansible-haproxy_server$ ansible-playbook test.yml -l 172.20.41.124

user@host:~/openstack-ansible-haproxy_server$ ssh root@172.20.41.124

opensuse-423-haproxy-server:~ # (source /etc/os-release; echo ${PRETTY_NAME})
openSUSE Leap 42.3

opensuse-423-haproxy-server:~ # ps -o pid,euser,egroup,label,args -p $(pgrep -d, rsyslogd)
  PID EUSER EGROUP LABEL COMMAND
20852 root root unconfined /usr/sbin/rsyslogd -n

opensuse-423-haproxy-server:~ # ls -ldZ /var/log/haproxy{,/.}
drwxr-xr-x 2 haproxy adm ? 4096 Aug 1 04:50 /var/log/haproxy
drwxr-xr-x 2 haproxy adm ? 4096 Aug 1 04:50 /var/log/haproxy/.

opensuse-423-haproxy-server:~ # tail /var/log/haproxy/haproxy{,-status}.log
tail: cannot open '/var/log/haproxy/haproxy.log' for reading: No such file or directory
tail: cannot open '/var/log/haproxy/haproxy-status.log' for reading: No such file or directory

opensuse-423-haproxy-server:~ # date --rfc-3339=seconds
2018-08-01 04:52:57+00:00

opensuse-423-haproxy-server:~ # python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.INFO); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL0); logger.addHandler(syslog_handler); logger.info('host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log')"
opensuse-423-haproxy-server:~ # python -c "import logging, logging.handlers; logger = logging.getLogger(__name__); logger.setLevel(logging.WARN); syslog_handler = logging.handlers.SysLogHandler(facility=logging.handlers.SysLogHandler.LOG_LOCAL1); logger.addHandler(syslog_handler); logger.warn('host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log')"

opensuse-423-haproxy-server:~ # tail /var/log/haproxy/haproxy{,-status}.log
==> /var/log/haproxy/haproxy.log <==
2018-08-01T04:53:06.375857+00:00 host haproxy[1234]: local0.info -> /var/log/haproxy/haproxy.log

==> /var/log/haproxy/haproxy-status.log <==
2018-08-01T04:53:11.421188+00:00 host haproxy[1234]: local1.warn -> /var/log/haproxy/haproxy-status.log

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/587687

[Corey Wright (coreywright)]

Ansible playbook of and resulting log from testing haproxy's rsyslog configuration by generating syslog messages on ubuntu 16.04, ubuntu 18.04, opensuse 4.23, & centos 7 of commit 1e0aa6bf473e634d7f7c92561c3fbc3128ae2be3 (ie patch set 2 of https://review.openstack.org/587687).

https://gist.github.com/coreywright/c38b87608cd49cf47e12be87350208dd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/587687
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-haproxy_server/commit/?id=1e0aa6bf473e634d7f7c92561c3fbc3128ae2be3
Submitter: Zuul
Branch: master

commit 1e0aa6bf473e634d7f7c92561c3fbc3128ae2be3
Author: Corey Wright <email address hidden>
Date: Tue Jul 31 04:47:01 2018 -0500

    Allow rsyslog to log HAProxy locally

    * Install haproxy-logging.cfg numerically before Ubuntu's
      /etc/rsyslog.d/49-haproxy.conf so its logging directives see HAProxy
      logs before they are discarded by 49-haproxy.conf.
    * Set owner of /var/log/haproxy to rsyslog's `syslog` user so rsyslog
      can write to it on Ubuntu.
    * Limit HAProxy-related rsyslog processing to HAProxy log messages
      instead of any/all log messages with the local0 or local1 facility
      and assuming HAProxy is the only application using those facilities.

    Change-Id: Ic259abc281619ba5ee8f020ac68373858a06e94d
    Closes-Bug: #1783886

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server yoga-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server xena-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-haproxy_server zed-eom release.