Servers-per-port can consume excessive OS threads

Bug #1554233 reported by Darrell Bishop
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Medium
Samuel Merritt

Bug Description

With servers-per-port enabled, the eventlet tpool thread pool size of 20 is excessive. For a servers-per-port deployment, a value of 1 vs. the default of 20 would be ideal.

Currently, an 84-disk chassis, with servers-per-port set to 4 would use up to 84 * 4 * 20 = 6720 OS threads for the kernel to juggle. With the 4 unix processes per disk, there's no real benefit for having more than one OS thread for eventlet tpool operations.

You can set the tpool thread pool size by calling `eventlet.tpool.set_num_threads()` any time before the first call to eventlet.tpool.execute().

The object server should either expose a tunable for the set_num_threads(N) value to be set (allowing swift-object-server config writers the ability to not screw up a deployment) or perhaps just pin it to 1 when servers-per-port is enabled in the config.

Changed in swift:
importance: Undecided → High
Changed in swift:
status: New → Confirmed
assignee: nobody → Samuel Merritt (torgomatic)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/289664

Changed in swift:
status: Confirmed → In Progress
clayg (clay-gerrard)
Changed in swift:
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/289664
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d9c4913e3b1aaba378d4786ddfefa6265c195f71
Submitter: Jenkins
Branch: master

commit d9c4913e3b1aaba378d4786ddfefa6265c195f71
Author: Samuel Merritt <email address hidden>
Date: Mon Mar 7 18:18:35 2016 -0800

    Make eventlet.tpool's thread count configurable in object server

    If you're running servers_per_port > 0 and threads_per_disk = 0 (as it
    should be with servers_per_port on), each object-server process will
    have 20 IO threads waiting around to service eventlet.tpool
    calls. This is far too many; with servers_per_port, there's no real
    benefit to having so many IO threads.

    This commit makes it so that, when servers_per_port > 0, each object
    server defaults to having one main thread and one IO thread.

    Also, eventlet's tpool size is now configurable via the object-server
    config file. If a tpool size is set, that's what we'll use regardless
    of servers_per_port. This allows operators with an excess of threads
    to remove some regardless of servers_per_port.

    Change-Id: I8f8914b7e70f2510393eb7c5e6be9708631ac027
    Closes-Bug: 1554233

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.15.0

This issue was fixed in the openstack/swift 2.15.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.