Random request to Contrail fails with 'Gateway time-out message'

Bug #1599156 reported by Oleksandr Savatieiev
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Hampapur Ajay

Bug Description

During performance tests on client environment we confirmed that Contrail is a bit slower for each object exists in DB. I.e. the more objects in system - the slower request served. When testing this, we confirmed at least 2 times slower request times with network create and list operations and network object count of 4000. So, we done some performance tuning to speed it up (Cassandra configs, memcache, etc.)

And in the process we experienced instability issue.
After couple of executions we experienced issue mentioned several times in Rally, and shortly after execution in CLI using any (!) command to neutron

It appears that Cassandra, at some point, becomes overloaded (IO operation) and fails to handle an operation and this is not handled in Contrail in any way and sent time-out message directly to user.

Environment and tools:
- KVM based MOS 8, HA with 3 Controllers and 3 Contrail nodes+DB
- FUEL Plugin v 4.0.0 for Contrail 3.0.2

Rally benchmarking tool’s “NeutronNetworks.create_and_list_networks” scenario was used with
- ‘constant' runner type
- concurrency of 20
- number of executions at 3000.

I.e. at the end of the test there is 3000 more network objects exists in the system.
To shorten test run time, we pre-created 1000 network objects using cli command:
 seq 1000 | parallel -j 5 "echo =========== creating network {}; neutron net-create {}" :::

description: updated
Illia Polliul (ipolliul)
affects: fuel-plugin-contrail → juniperopenstack
Changed in juniperopenstack:
assignee: nobody → Hampapur Ajay (hajay)
tags: added: config
removed: cassandra contrail
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.