Random request to Contrail fails with 'Gateway time-out message'

[Oleksandr Savatieiev]

During performance tests on client environment we confirmed that Contrail is a bit slower for each object exists in DB. I.e. the more objects in system - the slower request served. When testing this, we confirmed at least 2 times slower request times with network create and list operations and network object count of 4000. So, we done some performance tuning to speed it up (Cassandra configs, memcache, etc.)

And in the process we experienced instability issue.
After couple of executions we experienced issue mentioned several times in Rally, and shortly after execution in CLI using any (!) command to neutron

It appears that Cassandra, at some point, becomes overloaded (IO operation) and fails to handle an operation and this is not handled in Contrail in any way and sent time-out message directly to user.

Environment and tools:
- KVM based MOS 8, HA with 3 Controllers and 3 Contrail nodes+DB
- FUEL Plugin v 4.0.0 for Contrail 3.0.2

Rally benchmarking tool’s “NeutronNetworks.create_and_list_networks” scenario was used with
- ‘constant' runner type
- concurrency of 20
- number of executions at 3000.

I.e. at the end of the test there is 3000 more network objects exists in the system.
To shorten test run time, we pre-created 1000 network objects using cli command:
 seq 1000 | parallel -j 5 "echo =========== creating network {}; neutron net-create {}" :::