Default task limit from systemd is too low for ceph

[Dr. Jens Harbott]

When started via systemd, there is a default limit of 512 Tasks, and it seems that each thread counts as a different task:

# systemctl status ceph-osd@2
* [e-mail] - Ceph object storage daemon
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2016-04-01 13:23:58 UTC; 12s ago
  Process: 31618 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id 2 --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
 Main PID: 31667 (ceph-osd)
    Tasks: 150 (limit: 512)
   CGroup: /system.slice/system-ceph\x2dosd.slice/[e-mail]
           `-31667 /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph

With larger clusters, ceph-osd may need more threads, so this limit should be raised in the service definition.

This may also affect ceph-mon or ceph-mds.

[Dr. Jens Harbott (j-harbott)]

I just found out that the limit is not applied after a reboot, it appears that this only happens when the service is (re-)started manually from the shell.

[James Page (james-page)]

I'm a bit baffled as to why this sometimes get applied, and sometime not.

[James Page (james-page)]

That said TaskAccounting is enabled by default for systemd, and the default limit is 512.

I suspect that we need to either set this high or to infinity...

[James Page (james-page)]

Patched and pull request made upstream as well:

https://github.com/ceph/ceph/pull/8450

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceph - 10.1.0-0ubuntu1

---------------
ceph (10.1.0-0ubuntu1) xenial; urgency=medium

  * New upstream release candidate for Ceph Jewel
    (see http://pad.lv/1563714 for FFe):
    - d/control,rules,librgw*: Add new binary packages for librgw2.
    - d/p/fix-systemd-escaping.patch,pybind-flags.patch: Dropped,
      included upstream.
    - d/p/*: Refresh remaining patches.
    - d/control: Add BD on libldap2-dev for rados gateway.
    - d/p/disable-openssl-linking.patch: Disable build time linking
      with OpenSSL due to licensing incompatibilities.
    - d/*.symbols: Add new symbols for RC.
    - d/python-*.install: Correct wildcards for python module install.
    - d/p/32bit-compat.patch: Cherry pick upstream fix for 32 bit
      compatibility, resolving FTBFS on armhf/i386.
  * d/rules: Strip rbd-mirror package correctly.
  * d/rules: Install upstart and systemd configurations for rbd-mirror.
  * d/copyright: Ensure that jerasure and gf-complete are not stripped
    from the upstream release tarball.
  * d/p/drop-user-group-osd-prestart.patch: Drop --setuser/--setgroup
    arguments from call to ceph-osd-prestart.sh; they are not supported
    and generate spurious non-fatal warning messages (LP: #1557461).
  * d/p/tasksmax-infinity.patch: Drop systemd limitation of number of
    processes and threads to long running ceph processes; the default
    of 512 tasks is way to low for even a modest Ceph cluster
    (LP: #1564917).
  * d/rules: Ensure that dh_systemd_start does not insert maintainer
    script snippets for ceph-mon and ceph-create-keys - service restart
    should be handled outside of the packaging as it is under upstart
    and for all other systemd unit files installed (LP: #1563330).

 -- James Page <email address hidden> Wed, 06 Apr 2016 09:17:59 +0100