Comment 2 for bug 1008720

Reviewed: https://review.openstack.org/8153
Committed: http://github.com/openstack/swift/commit/870ef0378f00bfee169d09b83c0f68df96a2eb0b
Submitter: Jenkins
Branch: milestone-proposed

commit 870ef0378f00bfee169d09b83c0f68df96a2eb0b
Author: Samuel Merritt <email address hidden>
Date: Fri Jun 1 15:24:21 2012 -0700

    Fix starvation in object server with fast clients.

    When an object server was handling concurrent GET or POST requests
    from very fast clients, it would starve other connected clients. The
    greenthreads responsible for servicing the fast clients would hog the
    processor and only rarely yield to another greenthread.

    The reason this happens for GET requests is found in
    eventlet.greenio.GreenSocket, in the send() method. When you call
    .send(data) on a GreenSocket, it immediately calls .send(data) on its
    underlying real socket (socket._socketobject). If the real socket
    accepts all the data, then GreenSocket.send() returns without yielding
    to another greenthread. Only if the real socket failed to accept all
    the data (either .send(data) < len(data) or by raising EWOULDBLOCK)
    does the GreenSocket yield control.

    Under most workloads, this isn't a problem. The TCP connection to
    client X can only consume data so quickly, and therefore the
    greenthread serving client X will frequently encounter a full socket
    buffer and yield control, so no clients starve. However, when there's
    a lot of contention for a single object from a large number of fast
    clients (e.g. on a LAN connected w/10Gb Ethernet), then one winds up
    in a situation where reading from the disk is slower than writing to
    the network, and so full socket buffers become rare, and therefore so
    do context switches. The end result is that many clients time out
    waiting for data.

    The situation for PUT requests is analogous; GreenSocket.recv() seldom
    encounters EWOULDBLOCK, so greenthreads seldom yield.

    This patch calls eventlet.sleep() to yield control after each chunk,
    preventing any one greenthread's IO from blocking the hub for very
    long.

    This code has the flaw that it will greenthread-switch twice when a
    send() or recv() does block, but since there isn't a way to find out
    if a switch occurred or not, there's no way to avoid it. Since
    greenlet switches are quite fast (faster than system calls, which the
    object server does a lot of), this shouldn't have a significant
    performance impact.

    Fixes LP#1008720

    Change-Id: I8549adfb4a198739b80979236c27b76df607eebf