ceilometer-expirer memory leak with sqlalchemy backend

[Francois Deppierraz]

This is the first time I'm running ceilometer-expirer because the database grew too big. Unfortunately, it crashes due to an out of memory error.

# dpkg -l ceilometer-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-====================================-=======================-=======================-==============================================================================
ii ceilometer-common 2014.1.2-0ubuntu1.1 all ceilometer common files
# grep time_to_live /etc/ceilometer/ceilometer.conf
time_to_live=2592000
# time ceilometer-expirer -v
2014-09-25 12:42:23.658 24989 WARNING ceilometer.openstack.common.db.sqlalchemy.session [-] This application has not enabled MySQL traditional mode, which means silent data corruption may occur. Please encourage the application developers to enable this mode.
Killed

real 53m0.672s
user 10m52.473s
sys 3m34.193s
# dmesg
[...]
[1132260.625571] Out of memory: Kill process 24989 (ceilometer-expi) score 708 or sacrifice child
[1132260.627454] Killed process 24989 (ceilometer-expi) total-vm:48635592kB, anon-rss:17555444kB, file-rss:1764kB
[...]

mysql> select count(*) from sample;
+----------+
| count(*) |
+----------+
| 11739310 |
+----------+
1 row in set (2.49 sec)

mysql>

[gordon chung (chungg)]

hi Francois, which version is this? and do you know roughly how many samples are being expired?

[Francois Deppierraz (francois-ctrlaltdel)]

Hi Gordon, this is running 2014.1.2 on Ubuntu 14.04.

I guess that around 8 millions samples out of the 11 millions in the database should have been expired.

[gordon chung (chungg)]

Francois, we made some improvements to the expirer in Juno so hopefully that issue was fixed. this is a bug previously opened regarding expirer performance: https://bugs.launchpad.net/ceilometer/+bug/1289166

there were inefficiencies in our model in the icehouse releases that we addressed. it's advisable to use Juno builds if you plan to use SQL backend.

[Francois Deppierraz (francois-ctrlaltdel)]

Hi Gordon, Thanks for your help, it indeed looks like a duplicate of the bug you mentionned.