Servers-per-port can consume excessive OS threads

Bug #1554233 reported by Darrell Bishop
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Medium
Samuel Merritt

Bug Description

With servers-per-port enabled, the eventlet tpool thread pool size of 20 is excessive. For a servers-per-port deployment, a value of 1 vs. the default of 20 would be ideal.

Currently, an 84-disk chassis, with servers-per-port set to 4 would use up to 84 * 4 * 20 = 6720 OS threads for the kernel to juggle. With the 4 unix processes per disk, there's no real benefit for having more than one OS thread for eventlet tpool operations.

You can set the tpool thread pool size by calling `eventlet.tpool.set_num_threads()` any time before the first call to eventlet.tpool.execute().

The object server should either expose a tunable for the set_num_threads(N) value to be set (allowing swift-object-server config writers the ability to not screw up a deployment) or perhaps just pin it to 1 when servers-per-port is enabled in the config.

Changed in swift:
importance: Undecided → High
Changed in swift:
status: New → Confirmed
assignee: nobody → Samuel Merritt (torgomatic)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (master)

Fix proposed to branch: master
Review: https://review.openstack.org/289664

Changed in swift:
status: Confirmed → In Progress
clayg (clay-gerrard)
Changed in swift:
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (master)

Reviewed: https://review.openstack.org/289664
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=d9c4913e3b1aaba378d4786ddfefa6265c195f71
Submitter: Jenkins
Branch: master

commit d9c4913e3b1aaba378d4786ddfefa6265c195f71
Author: Samuel Merritt <email address hidden>
Date: Mon Mar 7 18:18:35 2016 -0800

    Make eventlet.tpool's thread count configurable in object server

    If you're running servers_per_port > 0 and threads_per_disk = 0 (as it
    should be with servers_per_port on), each object-server process will
    have 20 IO threads waiting around to service eventlet.tpool
    calls. This is far too many; with servers_per_port, there's no real
    benefit to having so many IO threads.

    This commit makes it so that, when servers_per_port > 0, each object
    server defaults to having one main thread and one IO thread.

    Also, eventlet's tpool size is now configurable via the object-server
    config file. If a tpool size is set, that's what we'll use regardless
    of servers_per_port. This allows operators with an excess of threads
    to remove some regardless of servers_per_port.

    Change-Id: I8f8914b7e70f2510393eb7c5e6be9708631ac027
    Closes-Bug: 1554233

Changed in swift:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/swift 2.15.0

This issue was fixed in the openstack/swift 2.15.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers