We faced an issue when /var/lib/mongo partition is almost full, even when ceilometer configuration is set to store meters and events only from last two days:
# grep live /etc/ceilometer/ceilometer.conf
metering_time_to_live=172800
event_time_to_live=172800
however inside mongodb we're able to find older meters/events.
From logs which we have, it seems mongo is showing tons of errors like:
TTLMonitor] error processing ttl for db: ceilometer 0 assertion src/mongo/db/storage/extent_manager.cpp:109
I asked to provide further data and outputs from commands:
mongo
use ceilometer
show dbs
db.meter.dataSize()
db.event.dataSize()
db.resources.dataSize()
db.system.indexes.dataSize()
db.event.totalSize()
db.event.totalIndexSize()
db.event.storageSize()
db.meter.getIndexes()
db.stats()
db.meter.stats()
db.event.stats()
db.serverStatus()
db.getSiblingDB('admin').runCommand({getParameter: 1, ttlMonitorEnabled: 1})
db.event.find({"timestamp" : {$lt : ISODate("2019-05-25T01:20:00Z")}})
As I'm not sure if TTLMonitor is still enabled, but anyway it can suggest some strange crush of that.
I'm raising that ticket as I'm lack of other ideas which can help with further troubleshooting.
Thanks in advance for help here.
Best regards,
Jan
Fix proposed to branch: 9.0
Change author: Roman Lubianyi <email address hidden>
Review: https:/