iter_nodes not robust to eventlet trampolining

[Darrell Bishop]

The generator, swift.proxy.controllers.base.Controller.iter_nodes(), is used concurrently by swift.proxy.controllers.base.Controller.make_requests(). If iter_nodes() trampolines, as it will when sending the StatsD UDP packet if StatsD logging is configured, then another greenthread may try to use the generator resulting in an exception like "ValueError: generator already executing" and a 500 response from the proxy.

I first observed this with the probe tests against my SAIO when I had StatsD logging enabled. I got a stacktrace like this in the proxy-server log: http://paste.openstack.org/show/lS1UzUupEgsXjyfY1VkF/

The probe test failure itself looked like this: http://paste.openstack.org/show/1qdH3ACijK12qSNt2WSx/

This blog post discusses the problem and a potential solution: http://anandology.com/blog/using-iterators-and-generators/

[Samuel Merritt (torgomatic)]

Marking as high priority since it causes 500s.

(Also, I'm going to fix it, so I can mark it whatever I want. :P)

[Sergio Rubio (rubiojr)]

We have observed this too under high concurrency.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29125
Committed: http://github.com/openstack/swift/commit/bd3ca778d10c461210f26ea3d194cacc70bef975
Submitter: Jenkins
Branch: master

commit bd3ca778d10c461210f26ea3d194cacc70bef975
Author: Samuel Merritt <email address hidden>
Date: Tue May 14 13:04:01 2013 -0700

    Fix ValueError from handoff logging.

    If you have StatsD logging turned on, then the iterator returned by
    Ring.get_part_nodes() will emit StatsD packets when it yields a
    handoff node. That network IO may make eventlet trampoline to another
    greenthread before returning from next(). Now, if that other
    greenthread tries to call next() on that same iterator, it blows up
    with a ValueError.

    Any socket IO inside a generator's next() method can cause this. It's
    easiest to reproduce with StatsD logging turned on, but logging to
    syslog can trigger it too.

    You can see this happen sometimes in the proxy's make_requests method
    if two of the primary nodes are down. Greenthread A goes into next()
    to get a handoff node, then sends a StatsD packet, and so eventlet
    trampolines to Greenthread B. Now, Greenthread B also calls next() to
    get a handoff node, and dies with a ValueError.

    This commit wraps up concurrently-accessed iter_nodes() iterators in a
    new thing called a GreenthreadSafeIterator that serializes access.

    Bug 1180110
    Change-Id: I8fe13d7295c056a2cab9e084f5966078a49bdc13