Ceilometer sending very big messages to the InfluxDB, as result we have fault of heka:
2017/04/11 20:39:22 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/04/11 20:39:22 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/04/11 20:39:22 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/04/11 20:39:22 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/04/11 20:39:22 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded```
and
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
2017/04/11 20:51:10 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() ...usr/share/telemetry_lua_modules/common/resources.lua:124: strbuf output_limit exceeded
Next changes are fixing the problem.
- /etc/telemetry-collector-heka/global.toml
- /etc/telemetry-collector-heka/decoder-sample.toml
- /etc/telemetry-collector-heka/filter-influxdb_accumulator_sample.toml
- /etc/telemetry-collector-heka/encoder-resource-elasticsearch.toml
root@ic3-ctl01-scc:~# grep -r 'max_message_size' /etc/telemetry-collector-heka/
/etc/telemetry-collector-heka/global.toml:max_message_size = 524288
root@ic3-ctl01-scc:~# grep -r 'output_limit' /etc/telemetry-collector-heka/
/etc/telemetry-collector-heka/decoder-sample.toml:output_limit = 10000000
/etc/telemetry-collector-heka/filter-influxdb_accumulator_sample.toml:output_limit = 100000000
/etc/telemetry-collector-heka/encoder-resource-elasticsearch.toml:output_limit = 10000000
Please adjust default values for processing messages 350K and more.
