openstack_sample_amqp-sample_decoder can fail on certain payloads

Bug #1691894 reported by Collin M.
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StackLight
In Progress
Undecided
Collin M.

Bug Description

Certain samples appear to trigger a "strbuf output_limit exceeded" in heka. This error does not get cleanly handled, and causes the entire decoder to shut down.

This can lead to messages being unconsumed, and cause the metering.sample queue to grow excessively.

2017/05/18 22:18:55 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/05/18 22:18:55 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/05/18 22:18:55 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/05/18 22:18:55 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/05/18 22:18:55 Decoder 'openstack_sample_amqp-sample_decoder' error: FATAL: process_message() /usr/share/telemetry_lua_modules/common/samples.lua:187: strbuf output_limit exceeded
2017/05/18 22:18:55 Shutdown initiated.

It appears that this occurs to a call to cjson.encode without an exception handler.

Changed in lma-toolchain:
status: New → Confirmed
assignee: nobody → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
Collin M. (ziggit)
Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Collin M. (ziggit)
Collin M. (ziggit)
Changed in lma-toolchain:
status: Confirmed → Fix Committed
status: Fix Committed → In Progress
Revision history for this message
Collin M. (ziggit) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-plugin-openstack-telemetry (master)

Change abandoned by Clark Boylan (<email address hidden>) on branch: master
Review: https://review.opendev.org/468702
Reason: Fuel projects have been retired.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.