Linux security limits are not configured

[Denis Klepikov]

Environment:
- MOS 9.0
- Ubuntu

Security limits are not configured:

root@node-5:~# cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain> <type> <item> <value>
#
#Where:
#<domain> can be:
# - a user name
# - a group name, with @group syntax
# - the wildcard *, for default entry
# - the wildcard %, can be also used with %group syntax,
# for maxlogin limit
# - NOTE: group and wildcard limits are not applied to root.
# To apply a limit to the root user, <domain> must be
# the literal username root.
#
#<type> can have the two values:
# - "soft" for enforcing the soft limits
# - "hard" for enforcing hard limits
#
#<item> can be one of the following:
# - core - limits the core file size (KB)
# - data - max data size (KB)
# - fsize - maximum filesize (KB)
# - memlock - max locked-in-memory address space (KB)
# - nofile - max number of open files
# - rss - max resident set size (KB)
# - stack - max stack size (KB)
# - cpu - max CPU time (MIN)
# - nproc - max number of processes
# - as - address space limit (KB)
# - maxlogins - max number of logins for this user
# - maxsyslogins - max number of logins on the system
# - priority - the priority to run user process with
# - locks - max number of file locks the user can hold
# - sigpending - max number of pending signals
# - msgqueue - max memory used by POSIX message queues (bytes)
# - nice - max nice priority allowed to raise to values: [-20, 19]
# - rtprio - max realtime priority
# - chroot - change root to directory (Debian-specific)
#
#<domain> <type> <item> <value>
#

#* soft core 0
#root hard core 100000
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#ftp - chroot /ftp
#@student - maxlogins 4

# End of file

but https://github.com/openstack/fuel-library/commit/65eca547606edceedc15d8fadfd1e5822ba9f1f6

~# hiera roles
["ceph-osd", "compute"]

root@node-5:~# whoami
root
root@node-5:~# ulimit -n
1024

Related bug: https://bugs.launchpad.net/fuel/+bug/1429553

[Alexander Bozhenko (alexbozhenko)]

Denis, I checked in 8.0, and on computes content of limits.conf is the same empty default as you posted.
So is it needed on compute nodes? Init.d and init services do not care about limits.conf. So if we need rise limits, we need to do it for particular service (https://bugs.launchpad.net/fuel/+bug/1526992)

[Alexander Bozhenko (alexbozhenko)]

Also, on controller nodes limits for rabbitmq came from /etc/default

cat /proc/$(pidof /usr/lib/erlang/erts-7.1/bin/beam.smp)/limits | grep -e Limit -e files
Limit Soft Limit Hard Limit Units
Max open files 100000 100000 files

[Denis Klepikov (dklepikov)]

Limits should be increased on compute modes too. If not - instances will start failing reaching limits.

root@node-6:~# ps ax | grep 812448
 812448 ? Rl 60:33 /usr/bin/qemu-system-x86_64 -name instance-0000001a -S -machine pc-i440fx-vivid,accel=tcg,usb=off -m 64 -realtime
...
...oon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on

root@node-6:~# cat /proc/812448/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 31445 31445 processes
Max open files 1024 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 31445 31445 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

[Denis Klepikov (dklepikov)]

In case of using Ceph for volumes and ephemeral storage instances can cause resource allocation fail under heavy load. Instance will open different connection to all OSDs across Ceph cluster where parts of the it's volumes placed.
You are able to attach 10-20 volumes to 1 instance, format it and mount. But when you will try to save a lot of data to all attached volumes at the same time you will reach Max open files limits.

[Dmitry Teselkin (teselkin-d)]

Setting both hard / soft limits to 32768 should be enough [1]
Setting soft > hard have no sence since as soon as hard limit is reached you get error.
Setting soft < hard is also useless because we don't bother about quota management here.

To do so for a libvirt controlled by upstart:
* add to /etc/init/libvirtd.conf
  limit nofile 32768 32768
* restart libvirt service
* stop and start every VM to apply limits changes

[1] https://bugs.launchpad.net/fuel/+bug/1642941

[Oleksiy Molchanov (omolchanov)]

So, me and Denis made an investigation and decided to set 102400 for all slave nodes and 102400 for libvirt. M.Golub is okay with this decision too.

Preparing a patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/405355

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/409731

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/409735

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409731
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=06b8e87459343ac45527a6c833e94bc1f0f2d2a5
Submitter: Jenkins
Branch: master

commit 06b8e87459343ac45527a6c833e94bc1f0f2d2a5
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Nov 30 13:16:49 2016 +0200

    Refactor limits for max open files

    * Move mof limits to separate task
    * Add mof limits for non-controller nodes
    * Add mof limits for libvirt

    Closes-Bug: 1644318
    Change-Id: I56d041a16fc9ef4ec1853b6b8880974a5dd70757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/409735
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4b0f4ca5df2068c2508a8e298fd82a2b369ca7ff
Submitter: Jenkins
Branch: stable/newton

commit 4b0f4ca5df2068c2508a8e298fd82a2b369ca7ff
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Nov 30 13:16:49 2016 +0200

    Refactor limits for max open files

    * Move mof limits to separate task
    * Add mof limits for non-controller nodes
    * Add mof limits for libvirt

    Closes-Bug: 1644318
    Change-Id: I56d041a16fc9ef4ec1853b6b8880974a5dd70757

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/405355
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9b04335bf055803399b5544923d813ed75231aaf
Submitter: Jenkins
Branch: stable/mitaka

commit 9b04335bf055803399b5544923d813ed75231aaf
Author: Oleksiy Molchanov <email address hidden>
Date: Wed Nov 30 13:16:49 2016 +0200

    Refactor limits for max open files

    * Move mof limits to separate task
    * Add mof limits for non-controller nodes
    * Add mof limits for libvirt

    Closes-Bug: 1644318
    Change-Id: I56d041a16fc9ef4ec1853b6b8880974a5dd70757

[Ekaterina Shutova (eshutova)]

Checked on 9.2 snapshot #798, but fix doesn't eliminate the problem.

[Vladimir Kuklin (vkuklin)]

9.2 snapshot #798 did not contain the fix

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Alexey Stupnikov (astupnikov)]

This is a Won't fix bug for 6.0, 6.1 and 7.0 as it depends on bug #1429553. And bug #1429553 won't be fixed for those milestones.

[Alexey Stupnikov (astupnikov)]

IMO we shouldn't backport patch [1] as its changes internal architecture and uses new project [2] to make limits configurable and to enforce limits on non-controller nodes and libvirt processes.

Since there is a clear workaround (configure security limits manually), there is no need to backport this patch to stable/8.0. Changing stable/8.0 status to Won't Fix.

WA: configure security limits manually.

[1] https://review.openstack.org/405355
[2] https://github.com/fuel-infra/puppet-limits.git

[Ilya Bumarskov (ibumarskov)]

[root@nailgun ~]# seq 1 5 | xargs -tI{} ssh node-{} 'cat /etc/security/limits.conf | grep -v \#'
ssh node-1 cat /etc/security/limits.conf | grep -v \#
Warning: Permanently added 'node-1' (ECDSA) to the list of known hosts.

ssh node-2 cat /etc/security/limits.conf | grep -v \#
Warning: Permanently added 'node-2' (ECDSA) to the list of known hosts.

ssh node-3 cat /etc/security/limits.conf | grep -v \#
Warning: Permanently added 'node-3' (ECDSA) to the list of known hosts.

ssh node-4 cat /etc/security/limits.conf | grep -v \#
Warning: Permanently added 'node-4' (ECDSA) to the list of known hosts.

ssh node-5 cat /etc/security/limits.conf | grep -v \#
Warning: Permanently added 'node-5' (ECDSA) to the list of known hosts.

[root@nailgun ~]# cat /etc/fuel_build_number
1575