Analytics: Rate limit same type of messages going to Collector and log file

[Raj Reddy]

Today, there's no rate limit of messages going from Sandesh client to collector, we should have something in client so as not to overwhelm the collector

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13159
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13160
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13159
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13160
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13448
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13449
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13160
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13449
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13527
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13160
Committed: http://github.org/Juniper/contrail-sandesh/commit/fbe1e94e20cf28837e9e8fa8ec83f5c0a044e5c8
Submitter: Zuul
Branch: master

commit fbe1e94e20cf28837e9e8fa8ec83f5c0a044e5c8
Author: arvindvis <email address hidden>
Date: Tue Aug 18 15:45:54 2015 -0700

This fix adds sandesh systemlog message throttling support.
We are throttling the message on the send side with a circular
buffer of certail value. The size of the buffer would be the
messages/second that can be sent by the application.
If application sends more messages than the configured rate,
it will be dropped and a syslog message is recorded for each
overshoot.Following logic is added in the send part for both
c++ and python:

1. Record the current time in curr_time
2. If CB is full:
2.1. Compare time of the next free buffer with current time
2.2. If difference is more than a sec, then enque else drop
3. enqueue in CB

Currently the CB size cannot be changed midway
Partial-Bug: 1469414

Change-Id: I2817ed643f1cd02ca916332c0d4f09e3f7b5f2c5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13449
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13527
Committed: http://github.org/Juniper/contrail-sandesh/commit/56c4adc7ca923c43d112a55e3bbe872132695c89
Submitter: Zuul
Branch: R2.22-dev

commit 56c4adc7ca923c43d112a55e3bbe872132695c89
Author: arvindvis <email address hidden>
Date: Tue Aug 18 15:45:54 2015 -0700

This fix adds sandesh systemlog message throttling support.
We are throttling the message on the send side with a circular
buffer of certail value. The size of the buffer would be the
messages/second that can be sent by the application.
If application sends more messages than the configured rate,
it will be dropped and a syslog message is recorded for each
overshoot.Following logic is added in the send part for both
c++ and python:

1. Record the current time in curr_time
2. If CB is full:
2.1. Compare time of the next free buffer with current time
2.2. If difference is more than a sec, then enque else drop
3. enqueue in CB

Currently the CB size cannot be changed midway
Partial-Bug: 1469414

Change-Id: I2817ed643f1cd02ca916332c0d4f09e3f7b5f2c5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13671
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13671
Committed: http://github.org/Juniper/contrail-controller/commit/3a0ac65ab3838de817906016ad3263ed9a9c9c93
Submitter: Zuul
Branch: R2.22-dev

commit 3a0ac65ab3838de817906016ad3263ed9a9c9c93
Author: arvindvis <email address hidden>
Date: Mon Aug 31 14:46:21 2015 -0700

This fix introduces throtlling of sandesh messages. The daemons which
use sandesh generator can specify the throttling value in msgs/second
and specify it either as command line argument or in their respective
config file sandesh_send_rate_limit
Partial-Bug: 1469414

Change-Id: Idda249149f905a8a89e861e048e39b3f93f05ffd

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13766
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13449
Committed: http://github.org/Juniper/contrail-controller/commit/21ce99d9eeca4a06539fe47edef443d54e3a62fc
Submitter: Zuul
Branch: master

commit 21ce99d9eeca4a06539fe47edef443d54e3a62fc
Author: arvindvis <email address hidden>
Date: Mon Sep 7 12:51:00 2015 -0700

This fix introduces throtlling of sandesh messages. The daemons which
use sandesh generator can specify the throttling value in msgs/second
and specify it either as command line argument or in their respective
config file sandesh_send_rate_limit

Partial-Bug: 1469414

Change-Id: Idda249149f905a8a89e861e048e39b3f93f05ffd

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13766
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13766
Committed: http://github.org/Juniper/contrail-provisioning/commit/680117836d5f1a370e4dc03d5ef17df8c7147db6
Submitter: Zuul
Branch: R2.22-dev

commit 680117836d5f1a370e4dc03d5ef17df8c7147db6
Author: arvindvis <email address hidden>
Date: Fri Sep 11 12:44:24 2015 -0700

sandesh_send_rate_limit parameter added to config files
They are all commented now.

Change-Id: I9ff046bf863fde20b363e44d8de45fa940c91242
Closes-Bug: 1469414

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19049
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19053
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19053
Committed: http://github.org/Juniper/contrail-sandesh/commit/6ef149a85d24c5b55cbd7885065f6220c63d3588
Submitter: Zuul
Branch: R2.20

commit 6ef149a85d24c5b55cbd7885065f6220c63d3588
Author: arvindvis <email address hidden>
Date: Tue Aug 18 15:45:54 2015 -0700

This fix adds sandesh systemlog message throttling support.
We are throttling the message on the send side with a circular
buffer of certail value. The size of the buffer would be the
messages/second that can be sent by the application.
If application sends more messages than the configured rate,
it will be dropped and a syslog message is recorded for each
overshoot.Following logic is added in the send part for both
c++ and python:

1. Record the current time in curr_time
2. If CB is full:
2.1. Compare time of the next free buffer with current time
2.2. If difference is more than a sec, then enque else drop
3. enqueue in CB

Currently the CB size cannot be changed midway
Partial-Bug: 1469414

Conflicts:
 compiler/generate/t_cpp_generator.cc
 library/common/sandesh.sandesh
 library/common/sandesh_uve.sandesh
 library/cpp/sandesh.cc
 library/cpp/sandesh.h
 library/cpp/sandesh_statistics.cc
 library/cpp/test/sandesh_message_test.cc
 library/python/pysandesh/sandesh_stats.py

Change-Id: I2817ed643f1cd02ca916332c0d4f09e3f7b5f2c5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19049
Committed: http://github.org/Juniper/contrail-provisioning/commit/af03cb49590b7286824a62f9282900b9fdd9964f
Submitter: Zuul
Branch: R2.20

commit af03cb49590b7286824a62f9282900b9fdd9964f
Author: arvindvis <email address hidden>
Date: Fri Sep 11 12:44:24 2015 -0700

sandesh_send_rate_limit parameter added to config files
They are all commented now.

Change-Id: I9ff046bf863fde20b363e44d8de45fa940c91242
Closes-Bug: 1469414

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19110
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19149
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19150
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19151
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19170
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19173
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19110
Committed: http://github.org/Juniper/contrail-controller/commit/f46293cc7bba9cac37706749e78c69c056b6277c
Submitter: Zuul
Branch: R2.20

commit f46293cc7bba9cac37706749e78c69c056b6277c
Author: arvindvis <email address hidden>
Date: Mon Aug 31 14:46:21 2015 -0700

This fix introduces throtlling of sandesh messages. The daemons which
use sandesh generator can specify the throttling value in msgs/second
and specify it either as command line argument or in their respective
config file sandesh_send_rate_limit
Partial-Bug: 1469414

Conflicts:
 src/analytics/main.cc
 src/analytics/options.cc
 src/analytics/options.h
 src/analytics/test/options_test.cc
 src/config/api-server/utils.py
 src/config/device-manager/device_manager/device_manager.py
 src/config/schema-transformer/to_bgp.py
 src/config/svc-monitor/svc_monitor/svc_monitor.py
 src/control-node/main.cc
 src/control-node/options.h
 src/discovery/disc_server.py
 src/dns/test/dns_options_test.cc
 src/opserver/opserver.py
 src/query_engine/qed.cc
 src/storage/stats-daemon/stats_daemon/storage_nodemgr.py
 src/vnsw/agent/init/agent_param.cc
 src/vnsw/agent/init/agent_param.h

Change-Id: Idda249149f905a8a89e861e048e39b3f93f05ffd

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19151
Committed: http://github.org/Juniper/contrail-provisioning/commit/0c7f8d7dd1209fc89076ecb3d7c599fa2c069284
Submitter: Zuul
Branch: R2.22.x

commit 0c7f8d7dd1209fc89076ecb3d7c599fa2c069284
Author: arvindvis <email address hidden>
Date: Fri Sep 11 12:44:24 2015 -0700

sandesh_send_rate_limit parameter added to config files
They are all commented now.

Change-Id: I9ff046bf863fde20b363e44d8de45fa940c91242
Closes-Bug: 1469414

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19170
Committed: http://github.org/Juniper/contrail-sandesh/commit/086e7917d97633257845fd28c98ad8e68c6c90cf
Submitter: Zuul
Branch: R2.22.x

commit 086e7917d97633257845fd28c98ad8e68c6c90cf
Author: arvindvis <email address hidden>
Date: Tue Aug 18 15:45:54 2015 -0700

This fix adds sandesh systemlog message throttling support.
We are throttling the message on the send side with a circular
buffer of certail value. The size of the buffer would be the
messages/second that can be sent by the application.
If application sends more messages than the configured rate,
it will be dropped and a syslog message is recorded for each
overshoot.Following logic is added in the send part for both
c++ and python:

1. Record the current time in curr_time
2. If CB is full:
2.1. Compare time of the next free buffer with current time
2.2. If difference is more than a sec, then enque else drop
3. enqueue in CB

Currently the CB size cannot be changed midway
Partial-Bug: 1469414

Conflicts:
 compiler/generate/t_cpp_generator.cc
 library/common/sandesh.sandesh
 library/common/sandesh_uve.sandesh
 library/cpp/sandesh.cc
 library/cpp/sandesh.h
 library/cpp/sandesh_statistics.cc
 library/cpp/test/sandesh_message_test.cc
 library/python/pysandesh/sandesh_stats.py

Change-Id: I2817ed643f1cd02ca916332c0d4f09e3f7b5f2c5

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19185
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19185
Committed: http://github.org/Juniper/contrail-controller/commit/d155e9b546e18e7025598825c039a42cc53f892f
Submitter: Zuul
Branch: R2.22.x

commit d155e9b546e18e7025598825c039a42cc53f892f
Author: arvindvis <email address hidden>
Date: Mon Aug 31 14:46:21 2015 -0700

This fix introduces throtlling of sandesh messages. The daemons which
use sandesh generator can specify the throttling value in msgs/second
and specify it either as command line argument or in their respective
config file sandesh_send_rate_limit
Partial-Bug: 1469414

Conflicts:
 src/analytics/main.cc
 src/config/api-server/utils.py
 src/config/device-manager/device_manager/device_manager.py
 src/config/schema-transformer/to_bgp.py
 src/config/svc-monitor/svc_monitor/svc_monitor.py
 src/control-node/main.cc
 src/control-node/options.h
 src/discovery/disc_server.py
 src/dns/test/dns_options_test.cc
 src/opserver/opserver.py
 src/query_engine/qed.cc
 src/storage/stats-daemon/stats_daemon/storage_nodemgr.py
 src/vnsw/agent/init/agent_param.cc
 src/vnsw/agent/init/agent_param.h
Change-Id: Idda249149f905a8a89e861e048e39b3f93f05ffd

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19150
Committed: http://github.org/Juniper/contrail-sandesh/commit/b8c702a89d13881e93b9e215ae35853216b039d7
Submitter: Zuul
Branch: R2.21.x

commit b8c702a89d13881e93b9e215ae35853216b039d7
Author: arvindvis <email address hidden>
Date: Tue Aug 18 15:45:54 2015 -0700

This fix adds sandesh systemlog message throttling support.
We are throttling the message on the send side with a circular
buffer of certail value. The size of the buffer would be the
messages/second that can be sent by the application.
If application sends more messages than the configured rate,
it will be dropped and a syslog message is recorded for each
overshoot.Following logic is added in the send part for both
c++ and python:

1. Record the current time in curr_time
2. If CB is full:
2.1. Compare time of the next free buffer with current time
2.2. If difference is more than a sec, then enque else drop
3. enqueue in CB

Currently the CB size cannot be changed midway
Partial-Bug: 1469414

Conflicts:
 library/common/sandesh.sandesh
 library/common/sandesh_uve.sandesh
 library/cpp/sandesh_statistics.cc
 library/cpp/test/sandesh_message_test.cc
 library/python/pysandesh/sandesh_stats.py

Change-Id: I2817ed643f1cd02ca916332c0d4f09e3f7b5f2c5

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19149
Committed: http://github.org/Juniper/contrail-provisioning/commit/cf2ff214e8d96a11805ba580fad8ade913cf997c
Submitter: Zuul
Branch: R2.21.x

commit cf2ff214e8d96a11805ba580fad8ade913cf997c
Author: arvindvis <email address hidden>
Date: Fri Sep 11 12:44:24 2015 -0700

sandesh_send_rate_limit parameter added to config files
They are all commented now.

Change-Id: I9ff046bf863fde20b363e44d8de45fa940c91242
Closes-Bug: 1469414

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/19173
Submitter: Arvind (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/19173
Committed: http://github.org/Juniper/contrail-controller/commit/bcc9bbc62b93381e5b69ccb04ec2ce43c15d9bc7
Submitter: Zuul
Branch: R2.21.x

commit bcc9bbc62b93381e5b69ccb04ec2ce43c15d9bc7
Author: arvindvis <email address hidden>
Date: Thu Apr 14 16:51:02 2016 -0700

This fix introduces throtlling of sandesh messages. The daemons which
use sandesh generator can specify the throttling value in msgs/second
and specify it either as command line argument or in their respective
config file sandesh_send_rate_limit
Partial-Bug: 1469414
Change-Id: Idda249149f905a8a89e861e048e39b3f93f05ffd