Ceilometer-collector grows in memory under high load

[Alex Krzos]

I have found that the ceilometer-collector grows in RSS memory when put under high load. I have tested with both mongodb and gnocchi as a backend and both backends exhibited the issue with the collector so the issue appears to be with the collector.

To reproduce
1. Deploy ha-overcloud (3 controllers) with 2 compute nodes (tripleo deployed)
2. Tune nova allocation ratios to allow for more over-committing of the compute nodes (if needed for your hardware)
3. Tune ceilometer for backend gnocchi (If desired to see ceilometer-collector memory growth with gnocchi)
4. Tune ceilometer to poll more often (default polling is 600s, I have tested 5s, 10s, 60s)
5. Boot small instances on overcloud at rate of 20 every 1200s until you have 200 total instances

Actual results:
Ceilometer collector was witnessed growing in memory from a ~100MiB to over 5GiB and as high as 75GiB in some instances.

Attached are graphs of system performance metrics collected on the cloud during each test:

Test 1: 1 OSPd, 3 Controllers, 2 computes - mongo ceilometer backend, 10s interval, 200 instances booted
Test 2: 1 OSPd, 3 Controllers, 2 computes - gnocchi ceilometer backend, 10s interval, 200 instances booted
Test 3: 1 OSPd, 3 Controllers, 2 computes - gnocchi ceilometer backend, 60s interval, 200 instances booted