Flow processing optimizations

[Praveen]

Hi,

Im sharing some rough ideas on changes in flow module for quench. Let me know if you have any comments other ideas.

Vrouter Changes:
————————
The current vrouter can support upto 20,000 flows per second
Significant amount of time is spent in searching thru overflow table
On reducing over-flow table to 128, setup rate went to around 50,000
We will need vrouter to support higher numbers. One idea is to bunch the flow-setup messages
We send forward and reverse flows in different messages. At the minimum, we can have single message to add/delete both flows
Ksync Changes
———————
One simplest change can be to support bulk messaging in Ksync module. This should be simple change for us.

Agent Changes:
———————

Did some experiments on nodeg35 about flow setup. Here are the results,

I modified agent with following, to add 100,000 flows from pkt_handler module and add all flows with flow_index = -1. Here are the results,
Without any changes, it takes about 10 seconds to add last flow in Ksync module.
When I remove logging, Ksync and flow maintenance tree it takes about 3 seconds to add 100,000 flows.
Each of logging, ksync and flow-maintenance seem to add about 2 seconds to add 100,000 flows.
My ideas on improving the flow setup rates. All the tasks given below should be able to run in parallel.
Make flow setup run in multiple threads.
Add a work-queue for each thread
Use a hash of <src_port and dst_port> to identify thread processing flow. This works since in most of the cases (other than link-local flows) ports don’t change in forward and reverse flows.
Link-local flows are the only case where forward and reverse flows have different ports. We will force the link-local flows to run in thread-1.
How to identify link-local flow at earlier point? Maybe based on route-flags?
Flow tree per thread?
Run Ksync in a different task context
Flows from all thread context are added to single work-queue for Ksync processing
Run flow-management tree in a work-queue
Flow management module will be client to oper-tables
It will track oper-db to flow-entries mappings
On oper-db changes identifies flows to be updated and enqueues message to flow setup queues
Flow aging and stats collection is done in different task context
All logging and sandesh messages are handled in this context
Flow processing changes:
Currently, as part of flow processing we find attributes for both forward and reverse flows. I think this is making pkt_flow_info complicated. Maybe we can change it as follows,
Identify type of flow (layer-2/layer-3)
Do forward flow processing. The type of flow is identified along with this. The type of flows can be
Layer-2 Flow
Layer-3 Flow
IPv4 Flows
Simple Flow
Floating-IP flow
VRF Translate flow
Link-Local flows
Link-local host flows
IPv6 Flows
Simple Flow
Floating-IP flow
VRF Translate Flow
Link Local flows
Link-local host flows
Compute reverse flow
Compute ECMP attributes
Compute RPF NH
Add into flow-table
Notify Ksync queue
Notify Flow Management queue
Notify Sandesh Queue
Notify Aging/Stats collection module
Too many enqueues from flow module can potentially add to latency.
Combine few things link flow-management+sandesh+tracing in same task.
Maybe we can do multiple level of queues.
Regards,
Praveen

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12698
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12698
Committed: http://github.org/Juniper/contrail-controller/commit/0d26718d92d14b3282adde6a6deebca67836be8d
Submitter: Zuul
Branch: R2.22-dev

commit 0d26718d92d14b3282adde6a6deebca67836be8d
Author: Praveen K V <email address hidden>
Date: Wed Jul 29 16:08:06 2015 +0530

Flow optimizations - Run flow management in a work-queue

As part of flow processing we need to maintain data structures to keep
the flow action in-sync with config changes. Building these changes and
also revaluating flows when config change is notified can result in
significant latencies.

With this change we move the flow management part to a work-queue. The
flow management module is resposible to keep the flow in-sync with
config changed.

Few other slow operations operations such as logging/UVE also will be
moved to this module in subsequent commits.

Change-Id: I2010e9256d908320c58d5bcc3ff66c2c638d398e
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12953
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12974
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12953
Committed: http://github.org/Juniper/contrail-controller/commit/535d7e97e48fd08e3d0154838b63f6d00c7992f6
Submitter: Zuul
Branch: R2.22-dev

commit 535d7e97e48fd08e3d0154838b63f6d00c7992f6
Author: Praveen K V <email address hidden>
Date: Fri Aug 7 09:18:25 2015 +0530

Split flow_table.cc to create new file flow_entry.cc

Move FlowEntry methods to new file flow_entry.cc and flow_entry.h
No changes in functionality

Change-Id: I5388e09b18bde3e631910c1f71bac03d4488103c
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/12992
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13030
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/12992
Committed: http://github.org/Juniper/contrail-controller/commit/b2ae967c8b464d54eab87b28534a2a6ad2cf1d62
Submitter: Zuul
Branch: R2.22-dev

commit b2ae967c8b464d54eab87b28534a2a6ad2cf1d62
Author: Ashok Singh <email address hidden>
Date: Tue Aug 11 23:24:45 2015 -0700

Move Flow logging to Flow Management module

Define a message to enqueue Flow Export requests in Flow Management module. Move FlowExport functionality from FlowTable to
Flow Management module. Replace FlowExport API calls in Flow Stats collector and Flow Table with a message to Flow Management
module.

Change-Id: I5bea219cac127f651d4c709f5e959779f89f3374
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13030
Committed: http://github.org/Juniper/contrail-controller/commit/a451af4ce4d3c84f788420b7e7d2015508597a65
Submitter: Zuul
Branch: R2.22-dev

commit a451af4ce4d3c84f788420b7e7d2015508597a65
Author: Praveen K V <email address hidden>
Date: Thu Aug 13 12:11:51 2015 +0530

Run FlowTable processing from work-queue

This change is a step towards running Flow setup in multiple threads. Flow
creation is a two step process,

FlowHandler :
FlowEntry are created and flow action are determined in this context.
This stage can potentially run in multiple threads (future commits)
FlowHandler runs from a workqueue in "Agent::FlowHandler" task context

FlowTable :
1. Manage flow_entry_map_ which contains all flows
2. Enforce the per-VM flow limits
3. Generate events to KSync and FlowMgmt modueles
FlowTable runs from a workqueue in "Agent::FlowTable" task context

Change-Id: I74c10561c20075a4fb5a3c2b09c1a8ddb9a9d599
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13173
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13173
Committed: http://github.org/Juniper/contrail-controller/commit/f148293fb1f803a7b60faae2159bbde0b7968404
Submitter: Zuul
Branch: R2.22-dev

commit f148293fb1f803a7b60faae2159bbde0b7968404
Author: Praveen K V <email address hidden>
Date: Mon Aug 17 21:43:09 2015 +0530

Optimize packet processing ASIO context

Method PktHandler::HandleRcvPkt is called from ASIO context. Following
processing was done in HandleRcvPkt within this context,

- Decode of the packet including decoding on tunnel headers
- In case of bare-metas identification of interface based on MAC address

This commit minimizes processing in ASIO context. Packet are enqueued to
module work-queue baesd on the agent-header. The packet decode is
subsequently done when work-queue is scheduled.

Change-Id: I658c74f25941a5b9c49c6ea84495f0477dca48d3
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13618
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13618
Committed: http://github.org/Juniper/contrail-controller/commit/f2456fc8baeab24e2ef3857d6a971d17daa455ef
Submitter: Zuul
Branch: R2.22-dev

commit f2456fc8baeab24e2ef3857d6a971d17daa455ef
Author: Praveen K V <email address hidden>
Date: Mon Sep 7 10:20:35 2015 +0530

Run packet-parse routins in ASIO context

In an earlier commit f148293fb1f803a7b60faae2159bbde0b7968404, the
packet-parse routines were moved out of ASIO context assuming it will
improve performance.

Further measurements show that, performing packet-parse in ASIO context
itself does not affect performance. Hence reverting the change.

Change-Id: Ifd6d8a5536b1e20314e2a314a4b57c7e1634ec09
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13975
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13975
Committed: http://github.org/Juniper/contrail-controller/commit/860f56196cf3d4a17eda7f2a94848f0b9ccb5182
Submitter: Zuul
Branch: R2.22-dev

commit 860f56196cf3d4a17eda7f2a94848f0b9ccb5182
Author: Praveen K V <email address hidden>
Date: Tue Sep 22 16:41:26 2015 +0530

Use variable sized buffer for KSync Encode

The old code was using 4K buffer for Sandesh Encode and KSync write
operations. However, most of the messages are much smaller.

The encoded message buffers are enqueued into async_send_queue_. In
scaled setups, the size of work-queue can grow large resulting in lots
of unused memory.

Modifying KSync to use 512bytes buffer by default. Only NH will use 4K
buffer to take care of large composite-NH.

Change-Id: I27408e6e96f8dea86056fb1760e4704851c4d4ae
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14391
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14411
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14391
Committed: http://github.org/Juniper/contrail-controller/commit/187cf8830648bcf061e471edd586d2b6186bdf9b
Submitter: Zuul
Branch: master

commit 187cf8830648bcf061e471edd586d2b6186bdf9b
Author: Praveen K V <email address hidden>
Date: Wed Jul 29 16:08:06 2015 +0530

Flow optimizations - Run flow management in a work-queue

As part of flow processing we need to maintain data structures to keep
the flow action in-sync with config changes. Building these changes and
also revaluating flows when config change is notified can result in
significant latencies.

With this change we move the flow management part to a work-queue. The
flow management module is resposible to keep the flow in-sync with
config changed.

Few other slow operations operations such as logging/UVE also will be
moved to this module in subsequent commits.

Partial-Bug: #1479295

Removing the config listener

After IFmap dependency manager is introduced for all the objects in Agent,
the config listener does not play any role other than invoking node observers
and link observers, which is taken care in dependency manager itself.

As part of the same, Uuid change of node is also detected and handled.

Couple of test cases are moved out of flaky tests.

closes-bug: #1480124

Split flow_table.cc to create new file flow_entry.cc

Move FlowEntry methods to new file flow_entry.cc and flow_entry.h
No changes in functionality

Partial-Bug: #1479295

Move Flow logging to Flow Management module

Define a message to enqueue Flow Export requests in Flow Management module. Move FlowExport functionality from FlowTable to
Flow Management module. Replace FlowExport API calls in Flow Stats collector and Flow Table with a message to Flow Management
module.

Partial-Bug: #1479295

Run FlowTable processing from work-queue

This change is a step towards running Flow setup in multiple threads. Flow
creation is a two step process,

FlowHandler :
FlowEntry are created and flow action are determined in this context.
This stage can potentially run in multiple threads (future commits)
FlowHandler runs from a workqueue in "Agent::FlowHandler" task context

FlowTable :
1. Manage flow_entry_map_ which contains all flows
2. Enforce the per-VM flow limits
3. Generate events to KSync and FlowMgmt modueles
FlowTable runs from a workqueue in "Agent::FlowTable" task context

Partial-Bug: #1479295

Optimize packet processing ASIO context

Method PktHandler::HandleRcvPkt is called from ASIO context. Following
processing was done in HandleRcvPkt within this context,

- Decode of the packet including decoding on tunnel headers
- In case of bare-metas identification of interface based on MAC address

This commit minimizes processing in ASIO context. Packet are enqueued to
module work-queue baesd on the agent-header. The packet decode is
subsequently done when work-queue is scheduled.

Partial-Bug: #1479295

* Track static and floating ip preference based on instance ip

1> Floating ip, static route and allowed address pair in ecmp
mode would have preference published based on instance-ip
preference
2> If allowed-address pair address is configured in active-stdby
mode, r...

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14411
Committed: http://github.org/Juniper/contrail-controller/commit/bc00ebff899116e73620e6d47445b4689f0a1634
Submitter: Zuul
Branch: master

commit bc00ebff899116e73620e6d47445b4689f0a1634
Author: Praveen K V <email address hidden>
Date: Tue Sep 22 16:41:26 2015 +0530

Use variable sized buffer for KSync Encode

The old code was using 4K buffer for Sandesh Encode and KSync write
operations. However, most of the messages are much smaller.

The encoded message buffers are enqueued into async_send_queue_. In
scaled setups, the size of work-queue can grow large resulting in lots
of unused memory.

Modifying KSync to use 512bytes buffer by default. Only NH will use 4K
buffer to take care of large composite-NH.

Change-Id: I27408e6e96f8dea86056fb1760e4704851c4d4ae
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14509
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14509
Committed: http://github.org/Juniper/contrail-controller/commit/6ee7df4473981a0fbae92c027b80df0bf8b85ed8
Submitter: Zuul
Branch: master

commit 6ee7df4473981a0fbae92c027b80df0bf8b85ed8
Author: Ashok Singh <email address hidden>
Date: Sun Oct 18 10:38:46 2015 -0700

Introspect changes for AclFlowReq and AclFlowCountReq

Define Per Ace Id flow count map as part of AclFlowMgmtEntry to make it part
of per ACL structure. This map maintains count of flows matching a given Ace ID.
This info is shown as part AclFlowCountReq. Remove AceIdFlowMgmtTree defined under
FlowMgmtManager as it is not required.
Also handle the callbacks for returning data for AclFlowReq
Partial-Bug: #1479295

Change-Id: Iba2fb146d3da38385864e4f84915b5f5015d3d0d

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14680
Submitter: Ashok Singh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14680
Committed: http://github.org/Juniper/contrail-controller/commit/dfb5d491485e8a124f25a8803904d2a143bc7640
Submitter: Zuul
Branch: master

commit dfb5d491485e8a124f25a8803904d2a143bc7640
Author: Ashok Singh <email address hidden>
Date: Wed Oct 28 02:23:40 2015 -0700

Move port bitmap updation to FlowStatsCollector Task from FlowMgmt task.

During flow add request, we use the source port and destination port from FlowKey and build a bitmap of
ports per vrouter, VN, VM and Interface. Move this code from FlowMgmt task to FlowStatsCollector Task
to mininmize the number of activities done in FlowMgmt task.

Change-Id: I98141ebbb4ef2925a85fd651163ee2370f6a2f00
Partial-Bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23691
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23691
Committed: http://github.org/Juniper/contrail-vrouter/commit/4e7721fbfc21061b21a76f402cb3202b00b51e78
Submitter: Zuul
Branch: master

commit 4e7721fbfc21061b21a76f402cb3202b00b51e78
Author: Divakar <email address hidden>
Date: Mon Aug 29 14:31:36 2016 +0530

Flow optimisations

1) Currently the flow request structure which sets the flow in Vrouter
contains IP address as array of bytes. This is to accomodate both V4 and
V6 IP addresses. It is observed that performance of Sandesh is lower for
array of bytes when compared to standard variable types. To avoid this,
array of bytes is replaced with 2 64 bit values. For IPV6 address both
the 64 bit values are used and for IPV4 addresses only 32 bits of lower 64
bit value is used. Bot source IP and destination IP are changed to use
the addresses like this.

2) When Vrouter traps the Forward flow to Agent, Agent first adds the
reverse flow and waits for the Ack from Vrouter to add the forward flow
again. This is to wait for the reverse flow index that is created by
Vrouter. This decreases the flow setup rate. Rather waiting for the
reverse flow index to be created by Vrouter, Agent writes the reverse
flow followed by forward flow with reverse flow key. Vrouter performs
a flow lookup using reverse flow key to retrieve reverse flow index.
This increases the flow setup rate as Agent is not waiting any more for the
reverse flow index to be generated by Vrouter. To accomodate this, flow
request structure is modified to contain reverse flow key in addition to
rever flow index. This key also uses IP address as 2 64 bit values.

3) The flow request structure is used right now to perform all the flow
operstaions in addition to flow setup. Like mapping the flow memory
initially by Agent, sending the flow response to Agent from Vrouter. Due
to this flow request structure contains lot many more fields than
required for setting the flow. So the flow request structure is trimmed
down to contain only the required fields for setting the flow. For flow
response from Vrouter to Agent another sandesh structure is created. And
flow_table_data is created to contain initial setup information. This
ensures both request and response structures contains the right fields.

4) Flow utility is modified to add large number of flows and to use
above new sandesh structures.

partial-bug: #1479295

Change-Id: I216ad9aa60bd0a9286ebb4c0173b323a4b7db97d

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24617
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24617
Committed: http://github.org/Juniper/contrail-vrouter/commit/43a0c7bd4c322688425e8138c8d9d56493035564
Submitter: Zuul
Branch: master

commit 43a0c7bd4c322688425e8138c8d9d56493035564
Author: Divakar D <email address hidden>
Date: Mon Oct 3 15:26:50 2016 +0530

Fix the compilation errors of uvtest of vrouter

The xml files are not modified or not properly modified to
run Vrouter tests after checking in https://review.opencontrail.org/#/c/23691/.
As per this change, flow ip addresses are not any more strings
and modified to two 64 bit values.

Now same is reflected in the vrouters tests too

Change-Id: I25100f84580671758858ade58afc5deec1118fb2
partial-bug: #1479295

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24723
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24723
Committed: http://github.org/Juniper/contrail-vrouter/commit/633d6a80b2c61a143efdcab6c50dd1cbc4728bbd
Submitter: Zuul
Branch: master

commit 633d6a80b2c61a143efdcab6c50dd1cbc4728bbd
Author: Divakar D <email address hidden>
Date: Thu Oct 6 16:38:58 2016 +0530

Enable Vrouter Vtests

Review request https://review.opencontrail.org/12698 has broken the
vrouter tests and the tests are disabled subsequently because of this.
Now the vr_flow_response sandesh message is added and the flow source ip
and destinatio ip is correctly added to xml file. This passes the tests.
The tests are reenabled too.

Change-Id: I99be64082eb3e85ea13529dd35cac14802cb3487
partial-bug: #1479295