Stop/Limit writing to cassandra from collector when disk full

[Raj Reddy]

Once the disk usage becomes high, collector should first stop writing flow samples into DB, if the disk continues to increase, at some point, collector should stop writing all messages to DB...

We could follow the same when the pending compaction tasks become high

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24918
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24918
Committed: http://github.org/Juniper/contrail-sandesh/commit/c728eb472da7c87cef520520808d4dba4a3a887e
Submitter: Zuul
Branch: master

commit c728eb472da7c87cef520520808d4dba4a3a887e
Author: Santosh Gupta <email address hidden>
Date: Fri Oct 14 15:31:53 2016 -0700

Stop/Limit writing to cassandra from collector when disk full

Sending sandesh repo changes. controller repo changes to follow after this is merged.
- Added HIGH_WATERMARK alarm
- 2 types of alarms are raised by alarmgen based on UVE
- HIGH_WATERMARK disk usage
- LOW_WATERMARK disk usage
- opserver queries db periodically
- kicks compaction task on db node if disk usage > LOW_WATERMARK
- sends disk-usage to collector
- collector decides on actions based on disk usage
- at HIGH_WATERMARK disk usage, stop writing all records to db
- at LOW_WATERMARK disk usage, stop writing all flow records to db
- HIGH/LOW watermark is configurable
- in analytics config file
- command-line
- set to default at 90% and 70%.
- disk_usage field on opserver/collector can be get/set from instropect
- Added automated UT for all above changes

Change-Id: I7f109d169680dd921c4279855fb053e463d56df2
Partial-Bug:1604200

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24961
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25416
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25417
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25416
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25417
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25416
Committed: http://github.org/Juniper/contrail-sandesh/commit/7e318ca811f62eebb9e7312157ef48c262b2e23c
Submitter: Zuul
Branch: master

commit 7e318ca811f62eebb9e7312157ef48c262b2e23c
Author: Santosh Gupta <email address hidden>
Date: Tue Oct 25 23:11:41 2016 -0700

Stop/Limit writing to cassandra from collector when disk full

Reverting previous commit due to design change.
This logic moved to controller repo.

Change-Id: I9816658b65b4371d5be076edc48bf2b2c7365137
Partial-Bug:1604200

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/25417
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/25417
Committed: http://github.org/Juniper/contrail-controller/commit/ee28287abb327d7e4b08bca071cca0d27f0838eb
Submitter: Zuul
Branch: master

commit ee28287abb327d7e4b08bca071cca0d27f0838eb
Author: Santosh Gupta <email address hidden>
Date: Mon Nov 21 17:15:09 2016 -0800

Stop/Limit writing to cassandra from collector when disk full

- separated watermark logic to separate lib. Used by
- queue_task (existing)
- disk-usage-percentage/pending-compaction-tasks (new)
- writing messages to collector is based on 3 parameters
- disk_usage on db nodes
- pending-compaction-tasks on db nodes
- severity level of incoming messages
- we configure disk-usage-percentage/pending-compaction-tasks watermarks map to
message severity-level on collector on startup
- this is configurable from collector config file or command line

- opserver fetches disk-usage-percentage and pending-compaction-tasks for all
db nodes periodically from uve
- opservers sends max disk-usage-percentage and max pending-compaction-tasks
amongst all db nodes to collector
- following values can be accessed via introspect
- disk-usage-percentage - get/set on opserver/collector
- pending-compaction-tasks - get/set on opserver/collector
- derived disk-usage-percentage-drop-level - get on collector
- derived pending-compaction-tasks-severity-level
- get on collector
- Added automated UT for all above changes using the above introspect api

Change-Id: Idb7703adb33fdb5c1d0a04330f3c190483427be5
Closes-Bug:1604200

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26405
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26406
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26405
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26406
Submitter: Santosh Gupta (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26405
Submitter: Santosh Gupta (<email address hidden>)