[fuel-library] Ceilometer's mongoDB is rapidly eating up partition space

[Michael Polenchuk]

Description of the environment:
Ceilometer with mongodb as backend

Steps to reproduce:
run in production environment

Expected result:
* time_to_live is configured ( N seconds > 0 )
* ceilometer-expirer is enabled to wipe out outdated metrics from database

Actual result:
ceilometer-expirer is disabled

Workaround:
Set time_to_live option in ceilometer.conf
and activate cron periodic expirer task

Impact:
Ceilometer collects quite huge amounts of metrics which keep in the database forever

In fuel 7.0 this should be done using native mongodb feature (as it's support was introduced in early kilo upstream code).

[Vladimir Kuklin (vkuklin)]

There is currently no best practice solution and we are too close to release of 6.0 right now. So we are putting this into release notes.

In order to apply a fix, user needs to setup cron job for ceilomter-expirer script and change time_to_live parameter to desired value in ceilometer.conf. After that he will need to restart collector service on all the controllers:

On CentOS:

/etc/init.d/openstack-ceilometer-collector restart

On Ubuntu:

/etc/init.d/ceilometer-collector restart

But user should be aware that all the old data will still be created with old ttl value, so if he needs he will need to wipe the data himself.

[Ivan Berezovskiy (iberezovskiy)]

It's impossible to choose right value ttl for ceilometer data, because it depends on ceilometer usage. For Fuel 6.1 we chose one week for storing data, for Fuel 7.0 we want to provide ability to set ttl from Web UI . Default value will be the same (one week), but it could be possible for users to change it.

[Dmitry Mescheryakov (dmitrymex)]

The bug is rather important, hence moving to Triaged for 6.0.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/155368

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148501
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4a05aa97b74097f8aae64389556175bdaeb52641
Submitter: Jenkins
Branch: master

commit 4a05aa97b74097f8aae64389556175bdaeb52641
Author: iberezovskiy <email address hidden>
Date: Tue Jan 20 14:13:51 2015 +0300

    Setup ttl for ceilometer data and cron daemon to cleanup

    * ttl is set to 1 week
    * cron job runs one a week

    Closes-bug: #1399164

    Change-Id: I5dc1ccc252602866d77ee44a3580842570401684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155368
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=34b163f2ab3ca8abf53367b53fd0589a1160da0a
Submitter: Jenkins
Branch: stable/6.0

commit 34b163f2ab3ca8abf53367b53fd0589a1160da0a
Author: iberezovskiy <email address hidden>
Date: Tue Jan 20 14:13:51 2015 +0300

    Setup ttl for ceilometer data and cron daemon to cleanup

    * ttl is set to 1 week
    * cron job runs one a week

    Closes-bug: #1399164

    Change-Id: I5dc1ccc252602866d77ee44a3580842570401684

[Vitaly Gusev (vgusev)]

Verified on ISO #521