Changing haproxy parameters with enabled SELinux after the initial deployment results in the haproxy container stuck in a restart loop

[Mike Fedosin]

Changing the haproxy configuration(e.g. timeouts adjustments or master/infra nodes scale out) after the initial deployment results in the haproxy container stuck in a restart loop due to selinux denials:

># docker ps | grep haproxy
ed4f116c597c 192.168.24.1:8787/rhosp14/openstack-haproxy:2018-12-07.1 "kolla_start" 11 minutes ago Restarting (1) 4 minutes ago haproxy

># docker logs -f haproxy
Running command: '/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'
+ echo 'Running command: '\''/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'\'''
+ exec /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg
<7>haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ds
[ALERT] 343/232417 (11) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/lib/haproxy/stats]
<5>haproxy-systemd-wrapper: exit, haproxy RC=1
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/haproxy/haproxy.cfg
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/haproxy/haproxy.cfg to /etc/haproxy/haproxy.cfg
INFO:__main__:Writing out command to execute
INFO:__main__:Setting permission for /var/lib/haproxy
ERROR:__main__:Failed to change ownership of /var/lib/haproxy to 42454:42454
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 345, in set_perms
    os.chown(path, uid, gid)
OSError: [Errno 13] Permission denied: '/var/lib/haproxy'
++ cat /run_command
+ CMD='/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
+ echo 'Running command: '\''/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'\'''
Running command: '/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg'
+ exec /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg
<7>haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ds
[ALERT] 343/232742 (11) : Starting frontend GLOBAL: error when trying to preserve previous UNIX socket [/var/lib/haproxy/stats]
<5>haproxy-systemd-wrapper: exit, haproxy RC=1

># grep denied /var/log/audit/audit.log
type=AVC msg=audit(1544502049.751:11633): avc: denied { setattr } for pid=52906 comm="python" name="haproxy" dev="vda2" ino=1022804 scontext=system_u:system_r:container_t:s0:c174,c325 tcontext=system_u:object_r:haproxy_var_lib_t:s0 tclass=dir permissive=0

Related bug description: https://bugzilla.redhat.com/show_bug.cgi?id=1658027

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/624373

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/625281

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624373
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=32f4db83c638af33225f2f2a58e141e535a619da
Submitter: Zuul
Branch: master

commit 32f4db83c638af33225f2f2a58e141e535a619da
Author: Mike Fedosin <email address hidden>
Date: Tue Dec 11 12:55:45 2018 +0100

    Fix access to /var/lib/haproxy when SELinux is enabled

    Currently we don't use relabeling of the folder when SELinux is enabled.
    This leads to the fact that we can not update the configuration of
    haproxy during the update, because of missing permissions.

    This commit adds the relabeling for the folder, which allows the
    container with haproxy to write into it.

    Closes-Bug: #1807933

    Change-Id: Ie79aed5f5665658ea09e000a4847062e9207e25c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625281
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1132612f7d8363cc4cf67515e9612bcb8d065cc0
Submitter: Zuul
Branch: stable/rocky

commit 1132612f7d8363cc4cf67515e9612bcb8d065cc0
Author: Mike Fedosin <email address hidden>
Date: Tue Dec 11 12:55:45 2018 +0100

    Fix access to /var/lib/haproxy when SELinux is enabled

    Currently we don't use relabeling of the folder when SELinux is enabled.
    This leads to the fact that we can not update the configuration of
    haproxy during the update, because of missing permissions.

    This commit adds the relabeling for the folder, which allows the
    container with haproxy to write into it.

    Closes-Bug: #1807933

    Change-Id: Ie79aed5f5665658ea09e000a4847062e9207e25c
    (cherry picked from commit 32f4db83c638af33225f2f2a58e141e535a619da)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.3.0 release.