heka failed parsing collectd

[tontinme]

We can not run lma_collector correctly on customer site. The environment is deployed with Fuel 7.0 and LMA is downloaded from Mirantis offical site.
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"

when I execute 'crm resource restart lma_collector', the /var/log/lma_collector.log show that:

2016/07/11 21:28:07 Input started: keystone_wsgi_logstreamer
2016/07/11 21:28:07 Input started: openstack_7_0_logstreamer
2016/07/11 21:28:07 Input started: openstack_warn_amqp
2016/07/11 21:28:07 Decoder 'keystone_7_0_logstreamer-keystone_7_0_decoder-1' error: Failed parsing: payload: l 9 17:30:20 node-37 keystone_wsgi_main_access 11.104.131.2 - - [10/Jul/2016:01:30:13 +0800] "OPTIONS / HTTP/1.0" 300 605 10901 "-" "-"
2016/07/11 21:28:07 Decoder 'openstack_logstreamer-openstack_decoder-1' error: Failed parsing: payload: Jul 10 00:51:50 node-37 cinder-api 2016-07-10 08:51:50.548 63303 INFO cinder.api.openstack.wsgi [-] http://11.104.131.2:8776/ returned with HTTP 200
2016/07/11 21:28:07 Decoder 'openstack_7_0_logstreamer-openstack_decoder-2' error: Failed parsing: payload: node-37 heat-engine 2016-07-10 04:02:45.242 114976 INFO heat.engine.service [req-04235052-0a34-4ee4-b463-ca74b249fbd5 - -] Service f6c86888-b115-47aa-86cc-96111f72fdae is updated
2016/07/11 21:28:07 Decoder 'openstack_logstreamer-openstack_decoder-3' error: Failed parsing: payload: node-37 heat-engine 2016-07-10 04:02:45.242 114976 INFO heat.engine.service [req-04235052-0a34-4ee4-b463-ca74b249fbd5 - -] Service f6c86888-b115-47aa-86cc-96111f72fdae is updated
2016/07/11 21:28:07 Decoder 'openstack_7_0_logstreamer-openstack_decoder-4' error: Failed parsing: payload: l 10 01:49:02 node-37 neutron-server 2016-07-10 09:49:02.301 20861 INFO neutron.wsgi [-] (20861) accepted ('11.104.131.2', 60050)
2016/07/11 21:28:07 Decoder 'openstack_logstreamer-openstack_decoder-4' error: Failed parsing: payload: l 9 17:30:20 node-37 keystone_wsgi_main_access 11.104.131.2 - - [10/Jul/2016:01:30:13 +0800] "OPTIONS / HTTP/1.0" 300 605 10901 "-" "-"

then I restart collectd 'service collectd restart', In /var/log/collectd.log:

[2016-07-11 21:47:01] haproxy: Mapping missing for "murano-api"
[2016-07-11 21:47:01] haproxy: Mapping missing for "murano_rabbitmq"
[2016-07-11 21:47:01] haproxy: Mapping missing for "murano_rabbitmq"
[2016-07-11 21:47:01] haproxy: Mapping missing for "murano_rabbitmq"
[2016-07-11 21:47:01] haproxy: Mapping missing for "murano_rabbitmq"
[2016-07-11 21:47:01] haproxy: Mapping missing for "murano_rabbitmq"
[2016-07-11 21:47:01] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2016-07-11 21:47:05] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2016-07-11 21:47:10] plugin_dispatch_values: Low water mark reached. Dropping 100% of metrics.
[2016-07-11 21:47:11] haproxy: Mapping missing for "murano-api"
[2016-07-11 21:47:11] haproxy: Mapping missing for "murano-api"

It seems that heka is still work without collectd. You can found lma_collector.log and screenshot of grafana in attachment.
(when i restart collectd, there are some volatility in grafana, as you can see in screenshot)

We notice the link https://bugs.launchpad.net/lma-toolchain/+bug/1517053, and patch the bug mentioned by the link, and nothing help.

[Swann Croiset (swann-w)]

Thanks for reporting bug.
what plugin (lma_collector) version do you have ? My guess is 0.8.0. if it is the case I invite you to upgrade to 0.8.1 which solves your issues. If it's a production environment and you cannot redeploy, we have a procedure which describes the upgrade.

you're hitting a caracteristic bug where heka is stucked and collectd starts to drop metrics: this is fixed with 0.8.1

[tontinme (tontinme-m)]

Thanks for your reply.
yes, this is a production environment. but unfortunately, I can not found the steps which describe the manual upgrade of lma_collector.

It will be a greatly appreciated if you can tell me here?

[Swann Croiset (swann-w)]

sorry for delay, I miss your last message, do you still need help here?