On a scaled flow setup, agent memory goes upto 24GB

[Vedamurthy Joshi]

R3.0.2.0 Build 26 Ubuntu 14.04 Kilo

On this setup, there were 100 vms sending hping traffic to every other VM.

A compute is configured with flow table size 4M. With active flow count around 1M to 1.7M, it is observed that agent memory goes upto 14GB

Per discussion with agent team, this could be optimized.

When flow_export_rate is set to 0 in global vrouter config, agent memory peaks at ~8GB

[Vedamurthy Joshi (vedujoshi)]

It is seen to goto 24GB as well

[Megh Bhatt (meghb)]

The sandesh client queue is limited to 100 KB for flow messages.

[Megh Bhatt (meghb)]

Also ran mock_generator which is a C++ generator that uses the sandesh client c++ lib to send flow messages as fast as it can and observed its memory usage and confirmed that it did not go above 125 MB.

[Ashok Singh (ashoksr)]

Observations in Vedu's setup which had TCP flows

Flow-export-rate – 100 (Default)
Max men usage was about 52G when active flow count was between 3400000 to 3700000
Mem usage was stable after the flows were decreasing. Observed that it was 52G for more than 20 minutes
Mem usage remained at 52G even when active flow count decreased from 3700000 to 2200000
Mem usage gradually increased from 41G to 47G over a period of 15 mins when the active flow count was in the range of 3000000 to 3500000 (add/deletes were happening with variable of about 500000 flows)

Similar behavior was observed when Flow-export-rate was disabled.

Flow-export-rate – 0
Max men usage was about 44G when flows were between 3400000 to 3600000

This was observed in 3.0.2.0-41~kilo (no private binary)
Mem usage was stable after the active flow count were decreasing. Observed that it was 44G for more than 20 minutes
Mem usage remained at 44G even when active flow count decreased from 3600000 to 2400000

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20738
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20738
Committed: http://github.org/Juniper/contrail-controller/commit/eb0dec8b4c5e46f53944f11b516d1b42b71389f7
Submitter: Zuul
Branch: R3.0

commit eb0dec8b4c5e46f53944f11b516d1b42b71389f7
Author: Praveen K V <email address hidden>
Date: Sat May 28 17:11:55 2016 +0530

Introspect enhancements to debug memory utilization

1. Update introspect to show max-flows and number of entries in
free-list
2. Fix for free-list grow request going to wrong flow-table

Change-Id: I631f6ec1535a07cc8cd12af6e7074eb5fe5f70cb
Partial-Bug: #1571163

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/20979
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/20979
Committed: http://github.org/Juniper/contrail-controller/commit/62d9b035e3d7c8e446beac7b37ddb4678f10671a
Submitter: Zuul
Branch: master

commit 62d9b035e3d7c8e446beac7b37ddb4678f10671a
Author: Praveen K V <email address hidden>
Date: Sat May 28 17:11:55 2016 +0530

Introspect enhancements to debug memory utilization

1. Update introspect to show max-flows and number of entries in
free-list
2. Fix for free-list grow request going to wrong flow-table

Conflicts:
 src/vnsw/agent/pkt/flow_event.h

Change-Id: I631f6ec1535a07cc8cd12af6e7074eb5fe5f70cb
Partial-Bug: #1571163

[Nagabhushana R (bhushana)]

commits so far are for enhancing some introspect content for better debuggability. We are in the process of repeating this scale test on latest 3.0.3 build and see where we are. Will update the bug with the readings.

[Hari Prasad Killi (haripk)]

Closing the bug as no memory leak was found. Depending on number of flows etc, the agent memory usage goes up and remains at that level. Further increase is only reported when more number of flows are setup.