timeouts pops in proxy during large object PUTs

[Alistair Coles]

It is possible for an entire object PUT data transfer to
execute without the greenthread sleeping and allowing other
greenthreads to run. This is more likely with an EC PUT because the
computation of EC fragments might be slower than the rate at which
they are drained out of IO send buffers, so IO never blocks. In
extreme cases this can cause timeouts in other greenthreads to pop, e.g.:

May 16 11:43:49 vagrant proxy-server: Timeout talking to memcached: 127.0.0.1:11211: with key_prefix XXXX, method incr, config_timeout 2.0, time_spent 16.21338200569153 (txn: txd42911c9952c48ddac72b-006463a4a4)

Note that the timeout does not pop when it's limit is reached - but after the concurrent PUT request completed (>16seconds). The timeout watcher greenthread is also starved by the PUT greenthread.

To reproduce the greedy greenthread behaviour on a VSAIO:

$ cat /etc/swift/proxy-server/proxy-server.conf.d/10_default.conf |grep workers workers = 1

$ dd if=/dev/zero of=1GB bs=4096 count=250000
$ swift post ecjunk -H 'X-Storage-Policy: ec'
$ time swift upload ecjunk 1GB
1GB
real    0m15.530s
user    0m2.362s
sys    0m0.400s

concurrently:

$ time swift stat
<snip>
real    0m14.326s
user    0m0.192s
sys    0m0.030s

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/swift/+/883367

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/swift/+/883367
Committed: https://opendev.org/openstack/swift/commit/aa96cb3dc675415f48d8bb2f8b61a4f394759f53
Submitter: "Zuul (22348)"
Branch: master

commit aa96cb3dc675415f48d8bb2f8b61a4f394759f53
Author: Alistair Coles <email address hidden>
Date: Wed May 17 12:54:45 2023 +0100

    proxy: add periodic zero-time sleep during object PUT

    Previously it was possible for an entire object PUT data transfer to
    execute without the greenthread sleeping and allowing other
    greenthreads to run. This was more likely with an EC PUT because the
    computation of EC fragments might be slower than the rate at which
    they are drained out of IO send buffers, so IO never blocks. In
    extreme cases this could cause timeouts in other greenthreads to pop.

    This patch adds a periodic zero-time sleep in the object PUT data
    transfer loop. An existing pattern in the GET path is re-used, and
    extracted to a new CooperativeIterator helper class.

    Change-Id: Idd6b767f1a746c72c106199f5d1fada3615b1e97
    Closes-Bug: #2019955
    Related-Change: Iae27109f5a3d109ad21ec9a972e39f22150f6dbb

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/swift 2.32.0 release.