percona cluster hits resource limits in HA Openstack cloud with xenial
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
percona-cluster (Juju Charms Collection) |
Won't Fix
|
Undecided
|
Unassigned | ||
percona-xtradb-cluster-5.6 (Ubuntu) |
Won't Fix
|
Undecided
|
Unassigned | ||
systemd (Debian) |
Fix Released
|
Unknown
|
|||
systemd (Ubuntu) |
Fix Released
|
High
|
Martin Pitt | ||
Xenial |
Fix Released
|
High
|
Martin Pitt |
Bug Description
I'm trying to deploy Mitaka Openstack using the 16.04 charms on Xenial using Juju 1.25.5 and MAAS 1.9.2, with as many components of Openstack being HA as possible.
When deployed, after running for a while mysql (which is a 3 node cluster) starts refusing connections, and erroring:
2016-05-03 01:25:28 13795 [ERROR] Error log throttle: 50 'Can't create thread to handle new connection' error(s) suppressed
When I look at systemd-cgtop, I can see it maxing out at 512 connections.
To get it going again I do a:
$ sudo systemctl edit mysql
and set:
TasksMax=infinity
Sometimes I even need to edit /etc/systemd/
I've noticed that dropping worker-multiplier setting on nova-cloud-
Please let me know if you need any more information.
SRU INFORMATION
---------------
Impact: Introducing a default #thread limit of 512 broke an unknown set of services which regularly run many threads.
Fix: http://
Regression potential: Very low -- this just restores the pre-228 behaviour and does not impose any new restriction.
Test case:
- Pick some unit like cron.service or mysql.service that does not specify an explicit TaskMax= limit.
- Check its TaskMax: systemctl show -p TasksMax cron.service
- In current xenial this is "512", after the update it should be a very big number (maxint minus 1, which means "infinity")
no longer affects: | percona-xtradb-cluster-5.6 (Ubuntu Xenial) |
Changed in systemd (Ubuntu Xenial): | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Martin Pitt (pitti) |
Changed in systemd (Debian): | |
status: | Unknown → Confirmed |
Changed in percona-cluster (Juju Charms Collection): | |
status: | New → Won't Fix |
description: | updated |
Changed in systemd (Debian): | |
status: | Confirmed → Fix Released |
Raising a package bug for this one; there is also an ongoing conversation about whether this limit should be dropped at the systemd level as an SRU, but that's still not decided AFAIK.