Linux security limits are not applied

[Andrey Shestakov]

Environment:
- MOS 6.0
- Ubuntu

Security limits are configured:
cat /etc/security/limits.conf
# Raising open file limit for OpenStack services
* soft nofile 102400
* hard nofile 112640

But limits are not applied:
# sudo -u rabbitmq bash
bash: /root/.bashrc: Permission denied
rabbitmq@node-23:~$ ulimit -n
1024
rabbitmq@node-23:~$

This affects all services.

For example, rabbitmq has internal limit for sockets count, which depends on linux nofile soft limit.
When sockets pool if fully used, new client's connections fails by timeout.

After investigation found that limits module is not used by PAM for su/sudo.
This line:
"session required pam_limits.so"
should be added to:
/etc/pam.d/common-session
/etc/pam.d/su
/etc/pam.d/sudo

[Bartłomiej Piotrowski (bpiotrowski)]

The ulimit settings for rabbitmq are set in /etc/default/rabbitmq-server. It can be easily proven that limits are applied:

root@node-1:~# pgrep -f -l rabbitmq
6738 rabbitmq-server
6756 su
6757 sh
6758 beam.smp
root@node-1:~# cat /proc/6738/limits | fgrep 'open files'
Max open files 102400 105472 files
root@node-1:~# cat /proc/6758/limits | fgrep 'open files'
Max open files 102400 105472 files

[Bartłomiej Piotrowski (bpiotrowski)]

My comment above applies to Ubuntu 14.04 and CentOS.

[Bartłomiej Piotrowski (bpiotrowski)]

Same goes for 6.0: see 827079a97bfd8ebcb58af241ccf3fbaf459953a9.

[Brad (asaint21)]

We're seeing the same issue.

root@node-10:/proc/5632# ps -ef | grep rabbitmq
rabbitmq 25899 1 0 Apr22 ? 00:00:05 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
root 30140 1 0 16:43 ? 00:00:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 30156 30140 0 16:43 ? 00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 30157 30156 0 16:43 ? 00:00:00 sh -c /usr/lib/rabbitmq/bin/rabbitmq-server
rabbitmq 30158 30157 46 16:43 ? 00:00:08 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -sname rabbit@node-10 -boot start_sasl -config /etc/rabbitmq/rabbitmq -rabbit tcp_listeners [{"auto",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-10-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-10"
rabbitmq 30420 30158 0 16:43 ? 00:00:00 inet_gethost 4
rabbitmq 30421 30420 0 16:43 ? 00:00:00 inet_gethost 4
root 32452 48387 0 16:43 pts/26 00:00:00 grep --color=auto rabbitmq
root@node-10:/proc/5632# grep 'open files' /proc/30158/limits
Max open files 1024 4096 files

[Bartłomiej Piotrowski (bpiotrowski)]

Could you please post here the output of 'fuel --fuel-version'?

[Brad (asaint21)]

We did some more troubleshooting and it looks like we've narrowed down the issue to the crm resource. When we start rabbitmq-server through init scripts, the appropriate ulimits get applied. When starting ( or restarting ) the resource with crm, the wrong ulimits are in place. It looks like this is because the crm resource is starting the rabbitmq-server binary directly instead of through the init script.

[root@fuel-master ~]# fuel --fuel-version
api: '1.0'
astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
auth_required: true
build_id: 2014-12-26_14-25-46
build_number: '58'
feature_groups:
- mirantis
- experimental
fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
production: docker
release: '6.0'
release_versions:
  2014.2-6.0:
    VERSION:
      api: '1.0'
      astute_sha: 16b252d93be6aaa73030b8100cf8c5ca6a970a91
      build_id: 2014-12-26_14-25-46
      build_number: '58'
      feature_groups:
      - mirantis
      fuellib_sha: fde8ba5e11a1acaf819d402c645c731af450aff0
      fuelmain_sha: 81d38d6f2903b5a8b4bee79ca45a54b76c1361b8
      nailgun_sha: 5f91157daa6798ff522ca9f6d34e7e135f150a90
      ostf_sha: a9afb68710d809570460c29d6c3293219d3624d4
      production: docker
      release: '6.0'

[Brad (asaint21)]

Or perhaps it's just inheriting the corosync process limits -- we added the ulimit settings from /etc/default/rabbitmq-server to /etc/default/corosync and booted the node and the appropriate limits were applied.

[Bartłomiej Piotrowski (bpiotrowski)]

Interestingly I cannot reproduce it on 6.1, even after reboot. I'll try again with 6.0

[root@node-16 ~]# pgrep -fl rabbitmq
31339 su rabbitmq -s /bin/sh -c /usr/sbin/rabbitmq-server >> "/var/log/rabbitmq/startup_log" 2>/dev/null
31343 sh -c /usr/sbin/rabbitmq-server >> "/var/log/rabbitmq/startup_log" 2>/dev/null
31345 /bin/sh /usr/sbin/rabbitmq-server
31351 /usr/lib64/erlang/erts-5.8.5/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../ebin -noshell -noinput -sname rabbit@node-16 -boot start_sasl -config /etc/rabbitmq/rabbitmq -rabbit tcp_listeners [{"auto",5673}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@node-16-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@node-16"
[root@node-16 ~]# fgrep 'open files' /proc/31351/limits
Max open files 102400 112640 files

[Brad (asaint21)]

Can you see what the results look like after restarting the resource with crm?

crm resource restart p_rabbitmq-server

[Bartłomiej Piotrowski (bpiotrowski)]

I have tested latest 6.0.1 ISO, restarted the resource with crm. Limits are applied correctly.

Please check newer iso, the commit I mentioned in #3 has been added on 2015-01-14.

[Brad (asaint21)]

When downloading from https://software.mirantis.com/openstack-downloads/, you are only provided with v6.0. Is 6.0.1 provided by a different channel?

[Bartłomiej Piotrowski (bpiotrowski)]

6.0.1 hasn't been released yet. Mirantis provides nightly builds for all releases in development here: https://ci.fuel-infra.org/

[krogon-intel (krogon-intel)]

Hello,
I have fuel 6.0.1 from 2015-05-12_14-36-59 and this issue exists.

root@node-157:~# cat /etc/security/limits.conf
# Raising open file limit for OpenStack services
* soft nofile 102400
* hard nofile 112640
root@node-157:~# HOME=/tmp sudo -u cinder bash
cinder@node-157:/root$ ulimit -n
1024

pam_limit.so is included in /etc/pam.d/su but does not exists in /etc/pam.d/sudo

[Bartłomiej Piotrowski (bpiotrowski)]

This bug report is about rabbitmq, I updated the title. Cinder should be tracked separately – although I don't think I've ever seen such issue or recommendation. Could you please open a new report with all information?

[krogon-intel (krogon-intel)]

Reading description rabbitmq was mentioned only as example. Let me quote: "This affects all services."

Indeed rabbitmq can work in 6.0.1 due to PAM module being included in /etc/pam.d/su but this do not resolve other services which uses sudo.

root@node-157:~# grep -R pam_limits.so /etc/pam.d
/etc/pam.d/cron:session required pam_limits.so
/etc/pam.d/sshd:session required pam_limits.so
/etc/pam.d/login:session required pam_limits.so
/etc/pam.d/su:# session required pam_limits.so

[Bartłomiej Piotrowski (bpiotrowski)]

Right, changed it back and re-opened.

[Armin ranjbar (zoup)]

Can confirm this in 6.0, apparently fixed in 6.1 .

[Alexander Bozhenko (alexbozhenko)]

Ulimits are not applied to services by design. Need to add them per-service if it is needed
https://bugs.launchpad.net/fuel/+bug/1526992

[Bogdan Dobrelya (bogdando)]

Added affected milestones. Additional description of the issue was provided in this duplicate https://bugs.launchpad.net/fuel/+bug/1526992

[Bogdan Dobrelya (bogdando)]

According to the comment #18, we shall describe which services we think have to be fixed to support increased ulimits

[Matthew Mosesohn (raytrac3r)]

This bug needs more info so we can concretely resolve it.

If we can't show that any service is in risk of failing due to low limits, we should move this to a medium priority tech debt because high priority should only be used for bugs that are actually impacting deployment.

[Matthew Mosesohn (raytrac3r)]

I did some research. I have some partial results about how many fds are open per service and their limits. I don't think we're in any major risks at the moment: https://etherpad.openstack.org/p/fuel-ubuntu-service-limits but I welcome input.

[Aleksey Zvyagintsev (azvyagintsev)]

This bug looks like bp-needed.
Should be discussed with core-team.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/270675

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270675
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=478fef32ceb706da22aff84ec28f54d03273c07e
Submitter: Jenkins
Branch: master

commit 478fef32ceb706da22aff84ec28f54d03273c07e
Author: Dmitry Bilunov <email address hidden>
Date: Thu Jan 21 12:07:59 2016 +0300

    Apply PAM security limits by running daemons through su

    Daemons launched from OCF scripts inherit default resource limits. It
    could confuse users and cause resource allocation fail under heavy load.

    We should run daemons requiring root privileges through su - root -c, so
    limits from limits.conf would be enforced. To make it easier to
    implement, a new wrapper "ocf_run_as_root" is implemented in
    ocf-fuel-funcs.

    Change-Id: I4b0e0cd0ebca9013adbd69830ea79772014d6cf1
    Closes-Bug: #1429553

[Bogdan Dobrelya (bogdando)]

This patch affects security by changing things like the haproxy start running as root user instead of the haproxy.

[Bogdan Dobrelya (bogdando)]

Revert https://review.openstack.org/#/c/273525/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/275275

[Bogdan Dobrelya (bogdando)]

The patch is OK and the comment #26 is wrong, sorry for a mistake made!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275275
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=65eca547606edceedc15d8fadfd1e5822ba9f1f6
Submitter: Jenkins
Branch: master

commit 65eca547606edceedc15d8fadfd1e5822ba9f1f6
Author: Dmitry Bilunov <email address hidden>
Date: Thu Jan 21 12:07:59 2016 +0300

    Apply PAM security limits by running daemons through su

    Daemons launched from OCF scripts inherit default resource limits. It
    could confuse users and cause resource allocation fail under heavy load.

    We should run daemons requiring root privileges through su - root -c, so
    limits from limits.conf would be enforced. To make it easier to
    implement, a new wrapper "ocf_run_as_root" is implemented in
    ocf-fuel-funcs.

    Change-Id: Iea56e4d08a2c1f92500129210d79e4b1fe04e3fd
    Closes-Bug: #1429553

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/275579

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/275579
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=464f57d51dd165f40a57121790e29df3f9616fc2
Submitter: Jenkins
Branch: stable/8.0

commit 464f57d51dd165f40a57121790e29df3f9616fc2
Author: Dmitry Bilunov <email address hidden>
Date: Thu Jan 21 12:07:59 2016 +0300

    Apply PAM security limits by running daemons through su

    Daemons launched from OCF scripts inherit default resource limits. It
    could confuse users and cause resource allocation fail under heavy load.

    We should run daemons requiring root privileges through su - root -c, so
    limits from limits.conf would be enforced. To make it easier to
    implement, a new wrapper "ocf_run_as_root" is implemented in
    ocf-fuel-funcs.

    Change-Id: Iea56e4d08a2c1f92500129210d79e4b1fe04e3fd
    Closes-Bug: #1429553
    (cherry picked from commit 65eca547606edceedc15d8fadfd1e5822ba9f1f6)

[Mikhail Samoylov (msamoylov)]

Verified on fuel version:
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Steps for verify:
1. Deploy cluster with 3 controllers
2. Check limits:
root@node-1:~# pidof beam.smp
18735
root@node-1:~# cat /proc/18735/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 23541 23541 processes
Max open files 102400 105472 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 23541 23541 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Values in /etc/security/limits.conf must be more than 4096

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/316085

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/316803

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/317979

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/317979

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/316803

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/316085

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.1- and 7.0-updates as this is not a bug but feature request.