Flow processing optimizations
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R2.20 |
Won't Fix
|
Medium
|
Praveen | |||
Trunk |
Fix Committed
|
Medium
|
Praveen |
Bug Description
Hi,
Im sharing some rough ideas on changes in flow module for quench. Let me know if you have any comments other ideas.
Vrouter Changes:
————————
The current vrouter can support upto 20,000 flows per second
Significant amount of time is spent in searching thru overflow table
On reducing over-flow table to 128, setup rate went to around 50,000
We will need vrouter to support higher numbers. One idea is to bunch the flow-setup messages
We send forward and reverse flows in different messages. At the minimum, we can have single message to add/delete both flows
Ksync Changes
———————
One simplest change can be to support bulk messaging in Ksync module. This should be simple change for us.
Agent Changes:
———————
Did some experiments on nodeg35 about flow setup. Here are the results,
I modified agent with following, to add 100,000 flows from pkt_handler module and add all flows with flow_index = -1. Here are the results,
Without any changes, it takes about 10 seconds to add last flow in Ksync module.
When I remove logging, Ksync and flow maintenance tree it takes about 3 seconds to add 100,000 flows.
Each of logging, ksync and flow-maintenance seem to add about 2 seconds to add 100,000 flows.
My ideas on improving the flow setup rates. All the tasks given below should be able to run in parallel.
Make flow setup run in multiple threads.
Add a work-queue for each thread
Use a hash of <src_port and dst_port> to identify thread processing flow. This works since in most of the cases (other than link-local flows) ports don’t change in forward and reverse flows.
Link-local flows are the only case where forward and reverse flows have different ports. We will force the link-local flows to run in thread-1.
How to identify link-local flow at earlier point? Maybe based on route-flags?
Flow tree per thread?
Run Ksync in a different task context
Flows from all thread context are added to single work-queue for Ksync processing
Run flow-management tree in a work-queue
Flow management module will be client to oper-tables
It will track oper-db to flow-entries mappings
On oper-db changes identifies flows to be updated and enqueues message to flow setup queues
Flow aging and stats collection is done in different task context
All logging and sandesh messages are handled in this context
Flow processing changes:
Currently, as part of flow processing we find attributes for both forward and reverse flows. I think this is making pkt_flow_info complicated. Maybe we can change it as follows,
Identify type of flow (layer-2/layer-3)
Do forward flow processing. The type of flow is identified along with this. The type of flows can be
Layer-2 Flow
Layer-3 Flow
IPv4 Flows
Simple Flow
Floating-IP flow
VRF Translate flow
Link-Local flows
Link-local host flows
IPv6 Flows
Simple Flow
Floating-IP flow
VRF Translate Flow
Link Local flows
Link-local host flows
Compute reverse flow
Compute ECMP attributes
Compute RPF NH
Add into flow-table
Notify Ksync queue
Notify Flow Management queue
Notify Sandesh Queue
Notify Aging/Stats collection module
Too many enqueues from flow module can potentially add to latency.
Combine few things link flow-management
Maybe we can do multiple level of queues.
Regards,
Praveen
Changed in juniperopenstack: | |
assignee: | nobody → Praveen (praveen-karadakal) |
tags: | added: vrouter |
Changed in juniperopenstack: | |
importance: | Undecided → Medium |
information type: | Proprietary → Public |
Review in progress for https:/ /review. opencontrail. org/12698
Submitter: Praveen K V (<email address hidden>)