ceilometer meter_time_to_live has no effect if applied to existing mongodb database

[Paul Gear]

If metering_time_to_live is left at its default (never expire data) and later switched to a positive value, it will have no effect. ceilometer implements expiry by using a mongodb TTL index. However, it does so when there is already an existing index on the timestamp field: https://pastebin.canonical.com/p/4vJNXQgDNM/

https://docs.mongodb.com/manual/core/index-ttl/ states "If a non-TTL single-field index already exists for a field, you cannot create a TTL index on the same field ... To change a non-TTL single-field index to a TTL index, you must drop the index first and recreate with the expireAfterSeconds option."

This index was added in the fix for lp:1193906; however those commits did so without taking into account the TTL index.

In the default configuration of ceilometer and ceilometer-charm, this isn't a problem, because data is never expired. But because the manual expiry of data is gated on whether TTL indexes are supported by the mongodb version in use (see _is_natively_ttl_supported in https://github.com/openstack/ceilometer/blob/e984f28b00e3a073c176f6435507f5bce1c82d30/ceilometer/storage/impl_mongodb.py), positive TTLs can result in data not being expired correctly.

Ideally, the code should be updated to create a TTL index if it is supported and a positive TTL is configured, and a non-TTL index if it is not supported (mongodb versions before 2.2), or TTL is non-positive.

Because mongodb is a deprecated storage driver for ceilometer, this isn't likely to be high priority or to be accepted by upstream, so I'll document the workaround here:

- Check if there's a TTL index on mongodb:

myset:PRIMARY> use ceilometer
switched to db ceilometer
myset:PRIMARY> db.meter.getIndexes()
...

- If an index called "meter_ttl" is present, ceilometer has configured a TTL index. Drop the non-TTL index:

myset:PRIMARY> db.meter.dropIndex("timestamp_idx")

Unfortunately, there's no indication in the mongodb documentation as to what will happen if "timestamp_idx" is recreated (which ceilometer seems to do on connection to mongodb). Hopefully the presence of the "meter_ttl" index should prevent "timestamp_idx" from breaking an existing TTL index, but I don't have any evidence to suggest one way or the other yet.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Corey Bryant (corey.bryant)]

Paul, thanks for reporting this. I've added upstream ceilometer as a target as well. For canonical bootstack I believe this affects deployments prior to queens, which is the point when charm deployments switched from mongodb to gnocchi.

[Erlon R. Cruz (sombrafam)]

After Queens, ceilometer has dropped support to MongoDB and removed the options mentioned in this bug. So we can now mark this bug as invalid.

[Erlon R. Cruz (sombrafam)]

https://review.opendev.org/#/c/512700/