Openstack HA , cmon monitoring script is not able to start mysql when mysql is down after few days.

[venu kolli]

Openstack HA , cmon monitoring script is not able to start mysql when mysql is down after few days.

Issue is observed on R2.0 build 22

More info to be added by ranjeet.

[venu kolli (vkolli)]

Assigning to ranjeet

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/6272
Committed: http://github.org/Juniper/contrail-provisioning/commit/da68ca883b64a60ce1ab0fbfcf669edddb321587
Submitter: Zuul
Branch: R2.1

commit da68ca883b64a60ce1ab0fbfcf669edddb321587
Author: Ranjeet R <email address hidden>
Date: Fri Jan 16 10:28:29 2015 -0800

Fixes: Openstack HA , cmon monitoring script is not able to start mysql when mysql is down after few days

Token cleanup and CMON logs cleanup is scheduled to run as a
cronjob in the midnight in all the three OpenStack controllers.
If the db is huge, it leads to WSREP lock issues leading to
data inconsistency. WSREP kills MySQL when there is data inconsistency
which leads to MySQL being killed.

To fix this, we will space out the cleanup job to run every hour.

Change-Id: I8088f18a06959eb2ef53416beb5b0bc29b44da00
Closes-Bug:1408756

[venu kolli (vkolli)]

fix needs to be revisited , marked it as blocker for 2.1

[Sanju Abraham (asanju)]

Fixed issue in the way cmon stats, logs and config get captured, stored and purged.

cmon logs all the states in DB that includes disk, cpu mem, table, transaction and monitoring events and logs. These logs will grow in size across the cluster when there are transactions happening on the mysql database.

The long term fix address this by increasing the time to collect and reducing the purge duration and truncating the un-reported statistics, alarms and logs which are not consumed by any application and not notified.

Fix is committed as part of https://review.opencontrail.org/#/c/7398

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7493
Committed: http://github.org/Juniper/contrail-provisioning/commit/94be5bdd4c4f7c384b7168ae784f71f25d2a08dd
Submitter: Zuul
Branch: R2.0

commit 94be5bdd4c4f7c384b7168ae784f71f25d2a08dd
Author: Ranjeet R <email address hidden>
Date: Mon Feb 16 12:15:15 2015 -0800

Fixes: This bug address the keystone token and cmon purge issue during scale and failure conditions

Galera cluster was having issues when DELETE calls where made to large tables like CMON.

This fix slows down the log collection and runs the command in only one node.

Change-Id: I77afc8854b8c6c284daa1b812666a57328dc7eca
Close-Bug:1408756

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7398
Committed: http://github.org/Juniper/contrail-provisioning/commit/c1563d8c9217943b0bed7e97d22d139e79040270
Submitter: Zuul
Branch: R2.1

commit c1563d8c9217943b0bed7e97d22d139e79040270
Author: Sanju Abraham <email address hidden>
Date: Thu Feb 12 19:39:51 2015 -0800

Close-Bug:1408756. This bug address the keystone token and cmon purge issue during scale and failure conditions

Change-Id: I9429bd75315b165f8b28592723de60da92a137ea