Heka decoder and filter metrics aren't collected

[Simon Pasquier]

This is a follow up of https://bugs.launchpad.net/lma-toolchain/+bug/1545743.

It can happen that the message injected by the heka_monitoring filter exceeds the maximum message size limit. A typical scenario is when the logstreamers match hundreds of files because Heka will create 1 decoder per file.
In that situation, no Heka metric are emitted and the related Grafana panels show no data.

There are (at least) 2 options to fix this issue:
- Increase the max message size limit.
- Strip down the number of collected metrics.

[Swann Croiset (swann-w)]

A third option would be to split the "bulk metric" message in smaller pieces:
https://github.com/openstack/fuel-plugin-lma-collector/blob/master/deployment_scripts/puppet/modules/lma_collector/files/plugins/filters/heka_monitoring.lua#L83

[Simon Pasquier (simon-pasquier)]

In that case, we would be limited by the max_process_inject parameter.

Anyway, it would make more sense to aggregate the metrics by classes of decoder because right now the data is barely usable (too many decoder items). My take is that we should rethink which metrics we collect (for instance it would be useful to collect InChanCapacity/InChanLength) and how we aggregate them for efficiency and usability.

[Swann Croiset (swann-w)]

yes we must collect InChanCapacity/InChanLenth metrics.

the max_process_inject is set to 10 and I'm confident this is enough to split in 10 pieces the "bulk metric". If it's not the case we can introduce the same mechanisum we have in GSE filter:
https://github.com/openstack/fuel-plugin-lma-collector/blob/master/deployment_scripts/puppet/modules/lma_collector/files/plugins/filters/gse_cluster_filter.lua#L113-L129