Redis require nofile set to at least 10032 to start correctly

[Luca Lorenzetto]

In a tripleo HA setup on RHEL 7 (custom built images), redis was not starting correctly and failing the cluster service.
After some debugging, limits value has been set to nofile 10032 (both soft/hard).
The service then has started correctly.

I think during image build or deployment a configuration in /etc/security/limits.d has to be inserted.

[Michele Baldessari (michele)]

Thanks Luca!

So we normally have these limits set in systemd config overrides:
[root@overcloud-controller-0 limits.d]# more /etc/systemd/system/redis.service.d/limit.conf [Service]
LimitNOFILE=10240

/etc/systemd/system/redis-sentinel.service.d/limit.conf
[Service]
LimitNOFILE=10240

But the problem is that we are currently using the OCF resource to manage redis master/slave via pacemaker. So those settings won't actually do anything (OCF does not invoke systemd). We do the same in puppetlabs-rabbitmq where the puppet module configures the limits in /etc/security/limits.d.

I see that the current puppet-redis module does not do that out of the box so we will need to:
A) Check if they can add that
B) Provide a workaround in the puppet-tripleo profile

[Michele Baldessari (michele)]

I submitted https://github.com/arioch/puppet-redis/issues/130 for point A) and will provide a workaround via B) shortly

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/389583

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/389583
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=03a68a5c8cdbabcf6e152b9aca62f1007c713b5c
Submitter: Jenkins
Branch: master

commit 03a68a5c8cdbabcf6e152b9aca62f1007c713b5c
Author: Michele Baldessari <email address hidden>
Date: Fri Oct 21 10:02:39 2016 +0200

    Set redis file descriptor limit when run via pacemaker

    The current redis file descriptor limit is 4096 because of two reasons:
    - It is run via the redis user
    - It is not started via systemd which has explicit LimitNOFILE set to
      10240 (which matches the default configuration of maximum 10000
      clients)

    Create an /etc/security/limits.d/redis.conf file in order to increase
    the fd limit value With this change we correctly get the following
    limits:
    [root@overcloud-controller-0 ~]# pcs status |grep -A2 redis
     Master/Slave Set: redis-master [redis]
         Masters: [ overcloud-controller-2 ]
         Slaves: [ overcloud-controller-0 overcloud-controller-1 ]

    [root@overcloud-controller-0 ~]# cat /proc/`pgrep redis`/limits | grep open
    Max open files 10240 10240 files

    Previously this limit was set to 4096.

    Change-Id: I7691581bad92ad9442cecd82cf44f5ac78ed169f
    Closes-Bug: #1635334

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/393343

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393343
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=9e9b64be7a8aa48acf349e3769fd7e93577d1969
Submitter: Jenkins
Branch: stable/newton

commit 9e9b64be7a8aa48acf349e3769fd7e93577d1969
Author: Michele Baldessari <email address hidden>
Date: Fri Oct 21 10:02:39 2016 +0200

    Set redis file descriptor limit when run via pacemaker

    The current redis file descriptor limit is 4096 because of two reasons:
    - It is run via the redis user
    - It is not started via systemd which has explicit LimitNOFILE set to
      10240 (which matches the default configuration of maximum 10000
      clients)

    Create an /etc/security/limits.d/redis.conf file in order to increase
    the fd limit value With this change we correctly get the following
    limits:
    [root@overcloud-controller-0 ~]# pcs status |grep -A2 redis
     Master/Slave Set: redis-master [redis]
         Masters: [ overcloud-controller-2 ]
         Slaves: [ overcloud-controller-0 overcloud-controller-1 ]

    [root@overcloud-controller-0 ~]# cat /proc/`pgrep redis`/limits | grep open
    Max open files 10240 10240 files

    Previously this limit was set to 4096.

    Change-Id: I7691581bad92ad9442cecd82cf44f5ac78ed169f
    Closes-Bug: #1635334
    (cherry picked from commit 03a68a5c8cdbabcf6e152b9aca62f1007c713b5c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 5.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 6.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 5.4.0 release.