commit 3ab2ada748a5f66c3f1dd717223a2d180f686c89
Author: Kevin Benton <email address hidden>
Date: Tue Feb 16 01:50:23 2016 -0800
Add exponential back-off RPC client
This adds an exponential backoff mechanism for timeout values
on any RPC calls in Neutron that don't explicitly request a timeout
value. This will prevent the clients from DDoSing the server by
giving up on requests and retrying them before they are fulfilled.
Each RPC call method in each namespace gets its own timeout value since
some calls are expected to be much more expensive than others and we
don't want to modify the timeouts of cheap calls.
The backoff currently has no reduction mechanism under the assumption
that timeouts not legitimately caused by heavy system load
(i.e. messages completely dropped by AMQP) are rare enough that the
cost of shrinking the timeout back down and potentially causing
another server timeout isn't worth it. The timeout does have a ceiling
of 10 times the configured default timeout value.
Whenever a timeout exception occurs, the client will also sleep for a
random value between 0 and the configured default timeout value to
introduce a splay across all of the agents that may be trying to
communicate with the server.
This patch is intended to be uninvasive for candidacy to be
back-ported. A larger refactor of delivering data to the agents
is being discussed in I3af200ad84483e6e1fe619d516ff20bc87041f7c.
Closes-Bug: #1554332
Change-Id: I923e415c1b8e9a431be89221c78c14f39c42c80f
(cherry picked from commit 3e668b6a3720c1509ffef4ad5b91b4242dfd47b3)
Reviewed: https:/ /review. openstack. org/314317 /git.openstack. org/cgit/ openstack/ neutron/ commit/ ?id=3ab2ada748a 5f66c3f1dd71722 3a2d180f686c89
Committed: https:/
Submitter: Jenkins
Branch: stable/mitaka
commit 3ab2ada748a5f66 c3f1dd717223a2d 180f686c89
Author: Kevin Benton <email address hidden>
Date: Tue Feb 16 01:50:23 2016 -0800
Add exponential back-off RPC client
This adds an exponential backoff mechanism for timeout values
on any RPC calls in Neutron that don't explicitly request a timeout
value. This will prevent the clients from DDoSing the server by
giving up on requests and retrying them before they are fulfilled.
Each RPC call method in each namespace gets its own timeout value since
some calls are expected to be much more expensive than others and we
don't want to modify the timeouts of cheap calls.
The backoff currently has no reduction mechanism under the assumption
that timeouts not legitimately caused by heavy system load
(i.e. messages completely dropped by AMQP) are rare enough that the
cost of shrinking the timeout back down and potentially causing
another server timeout isn't worth it. The timeout does have a ceiling
of 10 times the configured default timeout value.
Whenever a timeout exception occurs, the client will also sleep for a
random value between 0 and the configured default timeout value to
introduce a splay across all of the agents that may be trying to
communicate with the server.
This patch is intended to be uninvasive for candidacy to be 6e1fe619d516ff2 0bc87041f7c.
back-ported. A larger refactor of delivering data to the agents
is being discussed in I3af200ad84483e
Closes-Bug: #1554332 431be89221c78c1 4f39c42c80f 09ffef4ad5b91b4 242dfd47b3)
Change-Id: I923e415c1b8e9a
(cherry picked from commit 3e668b6a3720c15