[library] Raise max file descriptors and process file limits for radosgw

[Denis Ipatov]

Radosgw will hang if it runs out of file descriptors. We should increase the limits to reduce this from happening again.

[root@node-37 ~]# /etc/init.d/ceph-radosgw start
Starting radosgw instance(s)...
runuser: cannot set user id: Resource temporarily unavailable
Starting client.radosgw.gateway... [FAILED]

The following action fix this problem:

vi /etc/security/limits.conf
apache soft nproc 2047
apache hard nproc 16384
apache soft nofile 2048
apache hard nofile 65536

And start radosgw.

[Dmitry Borodaenko (angdraug)]

Raised priority to High and retargeted for 5.1 since a High priority duplicate was raised.

[Dmitry Borodaenko (angdraug)]

Original bug description is incorrect, limits defined in /etc/security/limits.conf have no bearing on services started by sysvinit in CentOS or upstart in Ubuntu.

According to Debian Wiki, every service is responsible for setting its own limits:
https://wiki.debian.org/Limits

For example, in Debian (and Ubuntu) packages for Apache, limits can be set by uncommenting this line in /etc/apache2/envvars (honored by apachectl):
APACHE_ULIMIT_MAX_FILES='ulimit -n 65536'

A more generic solution on Ubuntu is to use "limit nofile" upstart option like this:
https://github.com/ceph/ceph/blob/firefly/src/upstart/radosgw.conf#L9

On CentOS, there's no corresponding option since the init script doesn't use apachectl. Corresponding ulimit commands will have to be patched directly into the init script in the httpd package. Likewise for Ceph (including RadosGW) on CentOS, upstream init scripts include hardcoded ulimit commands:
https://github.com/ceph/ceph/blob/firefly/src/init-radosgw.sysv#L83

[Dmitry Borodaenko (angdraug)]

Observation of limits reported for Apache and RadosGW on MOS 5.1 doesn't match the values defined in their sysvinit and upstart init scripts.

CentOS sysvinit script for RadosGW:
daemon --user="$user" "ulimit -n 32768; $RADOSGW -n $name"

Actual soft/hard limit is 1024/4096 for RadosGW:
Max open files 1024 4096 files

...and 1024/1024 for Apache (init script doesn't set limits):
Max open files 1024 1024 files

Ubuntu upstart script for RadosGW (sysvinit script in the same deb does not set limits):
limit nofile 8096 65536

Actual soft/hard limit is 1024/4096 (same as CentOS).

...and 8192/8192 for Apache (init script doesn't set limits):
Max open files 8192 8192 files

In HA deployments, /etc/security/limits.conf is set (by corosync Puppet module) to:
* soft nofile 102400
* hard nofile 112640

As expected based on the previous comment, these values have no effect on Apache and RadosGW.

[Sergii Golovatiuk (sgolovatiuk)]

I suppose the main reason that RadosGW shouldn't share the same user as Apache. As a sample we may switch from Apache to nginx, so patching can delete apache user.

On Centos radosgw shares the same user with httpd

On Ubuntu radosgw uses root
root 20207 0.0 0.2 1631600 5488 ? Ssl Sep09 0:52 /usr/bin/radosgw -n client.radosgw.gateway

which is quite bad for security reasons.

I think we should introduce radosgw user with own limits for CentOS and Ubuntu

[Aleksandr Didenko (adidenko)]

The problem here is caused by "nproc" ulimit, not "nofile". Since we use the same "apache" user for both httpd and rados-gw in CentOS, then on a powerfull hardware puppet would set high values for apache workers and wsgi processes/threads. So when you try to start any new process under "apache" user (like rados-gw), you will hit "nproc" ulimit right away:

# /etc/init.d/ceph-radosgw start
Starting radosgw instance(s)...
runuser: cannot set user id: Resource temporarily unavailable
Starting client.radosgw.gateway... [FAILED]
/usr/bin/radosgw is not running.

# su - apache
su: cannot set user id: Resource temporarily unavailable

This problem does not affect httpd itsef, because it sets high nproc value:

# grep processes /proc/$(pgrep -P 1 httpd)/limits
Max processes 14865 14865 processes

So I agree with Sergii - using a saparate user for rados-gw in ubuntu and centos would be the best solution here.

[Christopher Aedo (docaedo)]

On CentOS, without setting radogw to run as a different user, process limit can be raised by modifying /etc/security/limits.d/90-nproc.conf

]# grep processes /proc/$(pgrep -P 1 radosgw)/limits
Max processes 1024 22948 processes

# sed -i 's/1024/2048/' /etc/security/limits.d/90-nproc.conf

# /etc/init.d/ceph-radosgw restart
Starting client.radosgw.gateway... [ OK ]

# grep processes /proc/$(pgrep -P 1 radosgw)/limits
Max processes 2048 22948 processes

[Christopher Aedo (docaedo)]

Additionally regarding CentOS, the hard limit is set in the same file (/etc/security/limits.d/90-nproc.conf)

# cat 90-nproc.conf
* soft nproc 2048
* hard nproc unlimited
root soft nproc unlimited

# grep processes /proc/$(pgrep -P 1 radosgw)/limits
Max processes 2048 unlimited processes

[Christopher Aedo (docaedo)]

Under Ubuntu, changes in /etc/security/limits.conf only take effect after logging out and back in (or after restarting the system). To increase or remove the limit of number of processes for root user which radosgw runs under, create or adjust in /etc/security/limits.conf.

Set to 2001 in /etc/security/limits.conf, radosgw picks this up:
# grep process /proc/$(pgrep radosgw)/limits
Max processes 2001 unlimited processes

Change to unlimited, log out, log in, then restart radosgw
# sed -i 's/2001/unlimited/' /etc/security/limits.conf
# exit
logout, then log back in
# /etc/init.d/radosgw restart
Starting client.radosgw.gateway...
2014-10-06 18:10:32.871016 7f18e6e1a780 -1 WARNING: libcurl doesn't support curl_multi_wait()
2014-10-06 18:10:32.871022 7f18e6e1a780 -1 WARNING: cross zone / region transfer performance may be affected
/usr/bin/radosgw is running.
# grep process /proc/$(pgrep radosgw)/limits
Max processes unlimited unlimited processes

[Christopher Aedo (docaedo)]

Beware - the proposed resolution to this bug https://bugs.launchpad.net/fuel/+bug/1376564 (https://review.openstack.org/#/c/126254/) removes /etc/security/limits.d/90-nproc.conf under CentOS, which will impact this proposed resolution.

[Christopher Aedo (docaedo)]

Update - these changes for CentOS persist after reboot. They do NOT persist on Ubuntu, as upstart ignores the limits.conf file.

[Christopher Aedo (docaedo)]

On Ubuntu, in order to get this change to persist across reboots, radosgw init file needs an extra line, in start block.

# grep -B 3 -A 2 ulimit /etc/init.d/radosgw

case "$1" in
    start)
        ulimit -p 5123 #adjust max number of processes for radosgw
        for name in `ceph-conf --list-sections $PREFIX`;
        do

NOTE - since upstart uses dash, the ulimit flag to adjust number of processes (-p) is different than when run under bash (-u).

[Dmitry Borodaenko (angdraug)]

Related nproc problem is addressed in https://review.openstack.org/125971

[Matthew Mosesohn (raytrac3r)]

Is this still an open problem?

[Dmitry Borodaenko (angdraug)]

Good question. The nofile limit is still a problem, but the root cause of what caused the priority on this bug to be set to High turned out to be the nproc limit and that's now fixed. I'm still not happy with the way we handle these limits, but I don't think it's High priority anymore.

[Ryan Moe (rmoe)]

Medium priority bugs are not backported to stable releases.

[Pawel Stefanski (pejotes)]

For the 7.0 build we have
Max processes 11806 11806 processes
Max open files 1024 4096 files

VERSION:
  feature_groups:
    - experimental
  production: "docker"
  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "28"
  build_id: "2015-07-07_05-28-31"
  nailgun_sha: "d040c5cebc9cdd24ef20cb7ecf0a337039baddec"
  python-fuelclient_sha: "315d8bf991fbe7e2ab91abfc1f59b2f24fd92f45"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "08ca3e768ad0701db909002b291ff4cc1b435eed"
  fuel-ostf_sha: "a752c857deafd2629baf646b1b3188f02ff38084"
  fuelmain_sha: "4f2dff3bdc327858fa45bcc2853cfbceae68a40c"

[Roman Podoliaka (rpodolyaka)]

We no longer fix Medium bugs in 8.0, closing as Won't Fix

[Roman Podoliaka (rpodolyaka)]

Per https://bugs.launchpad.net/fuel/+bug/1331554/comments/14 and https://bugs.launchpad.net/fuel/+bug/1331554/comments/16 removing the release-notes tag, as the original problem was fixed (nprocs limit). It's not clear if we need to do anything else.