timeouts pops in proxy during large object PUTs
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
High
|
Alistair Coles |
Bug Description
It is possible for an entire object PUT data transfer to
execute without the greenthread sleeping and allowing other
greenthreads to run. This is more likely with an EC PUT because the
computation of EC fragments might be slower than the rate at which
they are drained out of IO send buffers, so IO never blocks. In
extreme cases this can cause timeouts in other greenthreads to pop, e.g.:
May 16 11:43:49 vagrant proxy-server: Timeout talking to memcached: 127.0.0.1:11211: with key_prefix XXXX, method incr, config_timeout 2.0, time_spent 16.21338200569153 (txn: txd42911c9952c4
Note that the timeout does not pop when it's limit is reached - but after the concurrent PUT request completed (>16seconds). The timeout watcher greenthread is also starved by the PUT greenthread.
To reproduce the greedy greenthread behaviour on a VSAIO:
$ cat /etc/swift/
$ dd if=/dev/zero of=1GB bs=4096 count=250000
$ swift post ecjunk -H 'X-Storage-Policy: ec'
$ time swift upload ecjunk 1GB
1GB
real 0m15.530s
user 0m2.362s
sys 0m0.400s
concurrently:
$ time swift stat
<snip>
real 0m14.326s
user 0m0.192s
sys 0m0.030s
Changed in swift: | |
importance: | Undecided → High |
assignee: | nobody → Alistair Coles (alistair-coles) |
status: | New → In Progress |
Fix proposed to branch: master /review. opendev. org/c/openstack /swift/ +/883367
Review: https:/