percona cluster hits resource limits in HA Openstack cloud with xenial

[Brad Marshall]

I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible.

When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring:

  2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed

When I look at systemd-cgtop, I can see it maxing out at 512 connections.

To get it going again I do a:

  $ sudo systemctl edit mysql

and set:

  TasksMax=infinity

Sometimes I even need to edit /etc/systemd/system.conf and bump DefaultTasksMax to 1024 or higher, depending on long its been left running.

I've noticed that dropping worker-multiplier setting on nova-cloud-controller, neutron-api etc all help to reduce the load, but I still need to bump it up.

Please let me know if you need any more information.

SRU INFORMATION
---------------
Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads.
Fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0 (essentially, revert the upstream commit that enabled it)
Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction.
Test case:
 - Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit.
 - Check its TaskMax: systemctl show -p TasksMax cron.service
 - In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity")

[James Page (james-page)]

Raising a package bug for this one; there is also an ongoing conversation about whether this limit should be dropped at the systemd level as an SRU, but that's still not decided AFAIK.

[Brad Marshall (brad-marshall)]

FWIW I'm also seeing limits hit in other areas, rabbitmq-server seems to be a common one. I'm also randomly seeing:

  Failed to allocate directory watch: Too many open files

on the command line. I suspect its related to the percona and rabbitmq services hitting limits, but I'm not exactly sure.

[James Page (james-page)]

Re "Failed to allocate directory watch: Too many open files" message - I think that you might be on the right track, but it might not specifically be to the tasks limit systemd is imposing.

I have seen this message during testing on an alternative architecture, but assumed it was not related...

[James Page (james-page)]

Raising systemd task and assigning to Martin...

[Martin Pitt (pitti)]

This was discussed upstream last November: https://lists.freedesktop.org/archives/systemd-devel/2015-November/035006.html

And then enabled by default in 228 in https://github.com/systemd/systemd/commit/9ded9cd14.

So in retrospect, having a default limit there was not such a good idea after all: 512 is way too much for most "simple" services, and it's way too little for others such as the ones mentioned above. There is also no particular rationale about "512", so even if we'd bump it to 1024 we'd just make the limit even less useful while still breaking software.

So I think we should disable the default limit at least for Xenial in an SRU, but probably also in devel. It is both much safer and also much more effective in terms of guarding against berserk programs/bugs/unintended fork bombs etc. to set limits in units individually. Once someone looks at one, this is then a great time to also flip on the other resource and privilege limitations that systemd offers, such as CapabilityBoundingSet=, SecureBits=, PrivateDevices=, PrivateNetwork=, ProtectSystem=, ProtectHome=, etc.

[Martin Pitt (pitti)]

Upstream report: https://github.com/systemd/systemd/issues/3211

[James Page (james-page)]

+1 on your proposed approach Martin

[Martin Pitt (pitti)]

Debian unstable/yakkety fix: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=fe4d9d3ba0
Xenial backport: http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu-xenial&id=643089b1d

[Adam Conrad (adconrad)]

Hello Brad, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-6ubuntu1

---------------
systemd (229-6ubuntu1) yakkety; urgency=medium

  * Merge with Debian unstable. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.

systemd (229-6) unstable; urgency=medium

  * systemd-container: Prefer renamed "btrfs-progs" package name over
    "btrfs-tools". (Closes: #822629)
  * systemd-container: Recommend libnss-mymachines. (Closes: #822615)
  * Drop systemd-dbg, in favor of debhelpers' automatic -dbgsym packages.
  * Drop Add-targets-for-compatibility-with-Debian-insserv-sy.patch; we don't
    need $x-display-manager any more as most/all DMs ship native services, and
    $mail-transport-agent is not widely used (not even by our default MTA
    exim4).
  * Unify our two patches for Debian specific configuration files.
  * Drop udev-re-enable-mount-propagation-for-udevd.patch, i. e. run udevd in
    its own slave mount name space again. laptop-mode-tools 1.68 fixed the
    original bug (#762018), thus add a Breaks: to earlier versions.
  * Ship fbdev-blacklist.conf in /lib/modprobe.d/ instead of /etc/modprobe.d/;
    remove the conffile on upgrades.
  * Replace util-Add-hidden-suffixes-for-ucf.patch with patch that got
    committed upstream.
  * Replace Stop-syslog.socket-when-entering-emergency-mode.patch with patch
    that got committed upstream.
  * debian/udev.README.Debian: Adjust documentation of MAC based naming for
    USB network cards to the udev rule, where this was moved to in 229-5.
  * debian/extra/init-functions.d/40-systemd: Invoke status command with
    --no-pager, to avoid blocking scripts that call an init.d script with
    "status" with an unexpected pager process. (Closes: #765175, LP: #1576409)
  * Add debian/extra/rules/70-debian-uaccess.rules: Make FIDO U2F dongles
    accessible to the user session. This avoids having to install libu2f-host0
    (which isn't discoverable at all) to make those devices work.
    (LP: #1387908)
  * libnss-resolve: Enable systemd-resolved.service on package installation,
    as this package makes little sense without resolved.
  * Add a DHCP exit hook for pushing received NTP servers into timesyncd.
    (LP: #1578663)
  * debian/udev.postinst: Fix migration check from the old persistent-net
    generator to not apply to chroots. (Closes: #813141)
  * Revert "enable TasksMax= for all services by default, and set it to 512".
    Introducing a default limit on number of threads broke a lot of software
    which regularly needs more, such as MySQL and RabbitMQ, or services that
    spawn off an indefinite number of subtasks that are not in a scope, like
    LXC or cron. 512 is way too much for most "simple" services, and it's way
    too little for the ones mentioned above. Effective (and much stricter)
    limits should instead be put into units individually.
    (Closes: #823530, LP: #1578080)
  * Split out udev rule to name USB network interfaces by MAC address into
    73-usb-net-by-mac.rules, so that it's easier to disable. (Closes: #824025)
  * 73-usb-net-by-mac.rules: Disable when net.ifnames=0 is specified on the
    kernel comm...

[Brad Marshall (brad-marshall)]

I've upgraded systemd across my cluster to 229-4ubuntu6, removed all my custom tweaks to systemd settings, reloaded the daemon and both rabbitmq and mysql appear to be working fine on my openstack cluster. I'll be throwing a bit more load at it, but usually by this point mysql has fallen over, so I'd say this is a success.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu6

---------------
systemd (229-4ubuntu6) xenial-proposed; urgency=medium

  * Add a DHCP exit hook for pushing received NTP servers into timesyncd.
    (LP: #1578663)
  * Revert "enable TasksMax= for all services by default, and set it to 512".
    Introducing a default limit on number of threads broke a lot of software
    which regularly needs more, such as MySQL and RabbitMQ, or services that
    spawn off an indefinite number of subtasks that are not in a scope, like
    LXC or cron. 512 is way too much for most "simple" services, and it's way
    too little for the ones mentioned above. Effective (and much stricter)
    limits should instead be put into units individually.
    (Closes: #823530, LP: #1578080)
  * debian/gbp.conf: Switch to ubuntu-xenial branch.

 -- Martin Pitt <email address hidden> Thu, 12 May 2016 10:39:30 +0200

[Martin Pitt (pitti)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Waldemar Hamm (waldemarhamm)]

Will my systemd scripts with TasksMax setting keep working?

[Martin Pitt (pitti)]

> Will my systemd scripts with TasksMax setting keep working?

Yes, this only changes the builtin defaults if there is no explicit TaskMax= setting.

[Waldemar Hamm (waldemarhamm)]

Great to hear. This bug will be fixed then I guess: https://youtrack.jetbrains.com/issue/UP-6943

Thanks a lot!