Native OVSDB transation commit shows O(n) performance

[Ryan Moats]

Create 100 tenants each one with the following setup where each router is scheduled to the same legacy node that has the L3 agent configured to use the native OVSDB inerface.

tenant network ------- router ------ external network

Reference http://ibin.co/2GuI6plJvngR for graph of performance during set up of 100 routers.
In the above graph, y-axis is time in seconds, and x-axis is pass through _ovs_add_port (two per router add).

DbSetCommand's performance increases with each router add. To support scale, this needs to be closer to O(1) and perform significantly better than using ovs-vsctl via rootwrap daemon.

[Ryan Moats (rmoats)]

The above graph incorrectly assigns the transaction commit time to DbSetCommand. The corrected graph is here: http://ibin.co/2HEOucFKZBOv

[Ryan Moats (rmoats)]

Last graph for now: http://ibin.co/2HEkX59N4clf - this shows that the problem is within the commit_block code of the native OVSDB library that the native driver depends on itself.

[Ryan Moats (rmoats)]

Each tenant setup is done with the following neutron cli command set:

    neutron net-create network-$i --tenant-id $id
    neutron subnet-create network-$i 192.168.18.0/24 --name subnet-$i --tenant-id $id
    neutron router-create router-$i --tenant-id $id
    neutron router-gateway-set router-$i test-external
    neutron router-interface-add router-$i subnet-$i

where $i is the tenant number (runs from 1 to 100) and $id is the tenant-id for that tenant

[Ryan Moats (rmoats)]

This experiment was run in the following three node environment:

controller node
----------------------
running stock trusty tahr (3.13 kernel) + tip of tree devstack/openstack

network node
--------------------
running trusty tahr + 3.19 kernel + tip of tree devstack/openstack,
[ neutron code modified to instrument or profile the router create/update paths ]
configured to run OVS, DHCP, L3 (legacy) and metadata agents

compute node (not used in test, provided for completeness sake)
--------------------
running stock trusty tahr (3.13 kernel) + tip of tree devstack/openstack
configured to run nova-compute and OVS

Steps:
1. Create flat external network with default provider and assign a subnet to it on the controller.
2. Create 100 tenants via keystone on the controller.
For each tenant, run the commands in comment #3
3. Extract information from the instrumentation log from the L3 agent on the network node

[Ryan Moats (rmoats)]

Lowered the priority on this as it appears to have improved due to improvements in the OVS IDL. We may want to consider moving to using the OVS IDL directly to ensure that we are sharing the same pain points (and making the same performance improvements) as OVS.

[Ryan Moats (rmoats)]

scale is not a valid tag, please see: http://docs.openstack.org/developer/neutron/policies/bugs.html

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.