clients may be starved on fast networks

Bug #1008720 reported by John Dickinson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Fix Released
Undecided
Unassigned

Bug Description

very fast networks may starve clients on concurrent GETs

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to swift (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/8153

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to swift (milestone-proposed)

Reviewed: https://review.openstack.org/8153
Committed: http://github.com/openstack/swift/commit/870ef0378f00bfee169d09b83c0f68df96a2eb0b
Submitter: Jenkins
Branch: milestone-proposed

commit 870ef0378f00bfee169d09b83c0f68df96a2eb0b
Author: Samuel Merritt <email address hidden>
Date: Fri Jun 1 15:24:21 2012 -0700

    Fix starvation in object server with fast clients.

    When an object server was handling concurrent GET or POST requests
    from very fast clients, it would starve other connected clients. The
    greenthreads responsible for servicing the fast clients would hog the
    processor and only rarely yield to another greenthread.

    The reason this happens for GET requests is found in
    eventlet.greenio.GreenSocket, in the send() method. When you call
    .send(data) on a GreenSocket, it immediately calls .send(data) on its
    underlying real socket (socket._socketobject). If the real socket
    accepts all the data, then GreenSocket.send() returns without yielding
    to another greenthread. Only if the real socket failed to accept all
    the data (either .send(data) < len(data) or by raising EWOULDBLOCK)
    does the GreenSocket yield control.

    Under most workloads, this isn't a problem. The TCP connection to
    client X can only consume data so quickly, and therefore the
    greenthread serving client X will frequently encounter a full socket
    buffer and yield control, so no clients starve. However, when there's
    a lot of contention for a single object from a large number of fast
    clients (e.g. on a LAN connected w/10Gb Ethernet), then one winds up
    in a situation where reading from the disk is slower than writing to
    the network, and so full socket buffers become rare, and therefore so
    do context switches. The end result is that many clients time out
    waiting for data.

    The situation for PUT requests is analogous; GreenSocket.recv() seldom
    encounters EWOULDBLOCK, so greenthreads seldom yield.

    This patch calls eventlet.sleep() to yield control after each chunk,
    preventing any one greenthread's IO from blocking the hub for very
    long.

    This code has the flaw that it will greenthread-switch twice when a
    send() or recv() does block, but since there isn't a way to find out
    if a switch occurred or not, there's no way to avoid it. Since
    greenlet switches are quite fast (faster than system calls, which the
    object server does a lot of), this shouldn't have a significant
    performance impact.

    Fixes LP#1008720

    Change-Id: I8549adfb4a198739b80979236c27b76df607eebf

Thierry Carrez (ttx)
Changed in swift:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.