commit 3e668b6a3720c1509ffef4ad5b91b4242dfd47b3
Author: Kevin Benton <email address hidden>
Date: Tue Feb 16 01:50:23 2016 -0800
Add exponential back-off RPC client
This adds an exponential backoff mechanism for timeout values
on any RPC calls in Neutron that don't explicitly request a timeout
value. This will prevent the clients from DDoSing the server by
giving up on requests and retrying them before they are fulfilled.
Each RPC call method in each namespace gets its own timeout value since
some calls are expected to be much more expensive than others and we
don't want to modify the timeouts of cheap calls.
The backoff currently has no reduction mechanism under the assumption
that timeouts not legitimately caused by heavy system load
(i.e. messages completely dropped by AMQP) are rare enough that the
cost of shrinking the timeout back down and potentially causing
another server timeout isn't worth it. The timeout does have a ceiling
of 10 times the configured default timeout value.
Whenever a timeout exception occurs, the client will also sleep for a
random value between 0 and the configured default timeout value to
introduce a splay across all of the agents that may be trying to
communicate with the server.
This patch is intended to be uninvasive for candidacy to be
back-ported. A larger refactor of delivering data to the agents
is being discussed in I3af200ad84483e6e1fe619d516ff20bc87041f7c.
Reviewed: https:/ /review. openstack. org/280595 /git.openstack. org/cgit/ openstack/ neutron/ commit/ ?id=3e668b6a372 0c1509ffef4ad5b 91b4242dfd47b3
Committed: https:/
Submitter: Jenkins
Branch: master
commit 3e668b6a3720c15 09ffef4ad5b91b4 242dfd47b3
Author: Kevin Benton <email address hidden>
Date: Tue Feb 16 01:50:23 2016 -0800
Add exponential back-off RPC client
This adds an exponential backoff mechanism for timeout values
on any RPC calls in Neutron that don't explicitly request a timeout
value. This will prevent the clients from DDoSing the server by
giving up on requests and retrying them before they are fulfilled.
Each RPC call method in each namespace gets its own timeout value since
some calls are expected to be much more expensive than others and we
don't want to modify the timeouts of cheap calls.
The backoff currently has no reduction mechanism under the assumption
that timeouts not legitimately caused by heavy system load
(i.e. messages completely dropped by AMQP) are rare enough that the
cost of shrinking the timeout back down and potentially causing
another server timeout isn't worth it. The timeout does have a ceiling
of 10 times the configured default timeout value.
Whenever a timeout exception occurs, the client will also sleep for a
random value between 0 and the configured default timeout value to
introduce a splay across all of the agents that may be trying to
communicate with the server.
This patch is intended to be uninvasive for candidacy to be 6e1fe619d516ff2 0bc87041f7c.
back-ported. A larger refactor of delivering data to the agents
is being discussed in I3af200ad84483e
Closes-Bug: #1554332 431be89221c78c1 4f39c42c80f
Change-Id: I923e415c1b8e9a