Inconsistent Neutron deployments

[Justin Kilpatrick]

At the moment I'm not sure if this is an upstream/downstream issue so I'm filing it here, let me know if I should move it.

Description
=============================

I run automated performance tests on builds coming down the RDO pipeline. Recently some deployments of the same hash have terrible Neutron performance. If you follow the link and click through to Kibana you can see one example.

If you scroll down until you see errors for various neutron tests and look at the raw result distribution normal neutron API actions like create-port/router/subnet usually take between .5-2 seconds but we have some results taking more than 100 seconds, then the endpoint goes down and tests fail until it comes back up a few seconds later.

After redeploying the same RDO hash (the results on the top) performance is a much more normal distribution.

http://elk.browbeatproject.org/goto/410b4d31633bd3fc64875f4ae8960517

If this where a single event I would ignore it, but it's happened several times now, were it across multiple versions I would attribute it to some environmental conditions, but that's not the case either it's Ocata only.

dlrn_hash: 19e86b3bef7164ef4b15a28683e8e2c0165c9625_24809939
http://elk.browbeatproject.org/goto/48ead78ee235792dbd55618184c70027

dlrn_hash: dd2065a3901cbe205e357a3d6dfa822a2c1e45c2_d953e5b2
http://elk.browbeatproject.org/goto/b51c2f2b47de60047f9bd6e1c39c5151

dlrn_hash: 7c30084d848bb45f713f5dd74988725578334150_fa03eecc
http://elk.browbeatproject.org/goto/fbb2d33204ebe8de0af94c1f87ed08c6

dlrn_hash: bdc175be26655580004af41b1fd2cfe612ebf953_24809939
http://elk.browbeatproject.org/goto/53b45a064ad3157fc2776651e86c5689

dlrn_hash: 2bf63a3ea2cd6ce91efd85362c26a30943426c2e_08b50dba
http://elk.browbeatproject.org/goto/13e7520422bb0c06f3728993672193c9

Steps to reproduce
==========================

Deploy the latest drln hash from the RDO pipeline on baremetal and begin
creating/deleting several hundred neutron objects. Repeat until whatever
magic causes this bug triggers.

Expected Result
==========================

Neutron works as expected under load

Actual Result
=========================

Neutron is non-functional for non-trivial workloads

[Emilien Macchi (emilienm)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (FUTURE, PIKE, QUEENS, ROCKY, STEIN).
  Valid example: CONFIRMED FOR: FUTURE