Ceilometer

Inconsisten behaviour of ceilometrer kafka publisher

Bug #1628456 reported by raghunath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Undecided
celik esra

Bug Description

While using ceilometer kafka publisher with openstack mitaka,the bahaviour is inconsistent.
Initially the messages are published successfully.After few seconds they are failing with different errors.

Downloaded the latest kakfa verion no luck(1.3.2.dev) and using kafka versio kafka_2.11-0.9.0.0.tgz.

Please find the different errors that are seen in ceilometer-agent-notification.log:

2016-09-28 09:19:07.858 29051 ERROR kafka.producer [req-70bccdbc-e37a-43d3-9cdd-ba663db4f338 - - - - -] Unable to send messages
2016-09-28 09:19:07.858 29051 ERROR kafka.producer Traceback (most recent call last):
2016-09-28 09:19:07.858 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py", line 422, in _send_messages
2016-09-28 09:19:07.858 29051 ERROR kafka.producer fail_on_error=self.sync_fail_on_error
2016-09-28 09:19:07.858 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 648, in send_produce_request
2016-09-28 09:19:07.858 29051 ERROR kafka.producer (not fail_on_error or not self._raise_on_response_error(resp))]
2016-09-28 09:19:07.858 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 404, in _raise_on_response_error
2016-09-28 09:19:07.858 29051 ERROR kafka.producer raise resp
2016-09-28 09:19:07.858 29051 ERROR kafka.producer FailedPayloadsError: FailedPayloadsError
2016-09-28 09:19:07.858 29051 ERROR kafka.producer

-----------------------------------------------------------------------------------------------------------------------------------------
2016-09-28 09:19:07.860 29051 ERROR kafka.producer [req-17551213-3624-4aee-a083-b22be5e9c0e5 - - - - -] Unable to send messages
2016-09-28 09:19:07.860 29051 ERROR kafka.producer Traceback (most recent call last):
2016-09-28 09:19:07.860 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py", line 422, in _send_messages
2016-09-28 09:19:07.860 29051 ERROR kafka.producer fail_on_error=self.sync_fail_on_error
2016-09-28 09:19:07.860 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 644, in send_produce_request
2016-09-28 09:19:07.860 29051 ERROR kafka.producer resps = self._send_broker_aware_request(payloads, encoder, decoder)
2016-09-28 09:19:07.860 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 286, in _send_broker_aware_request
2016-09-28 09:19:07.860 29051 ERROR kafka.producer conn.recv()
2016-09-28 09:19:07.860 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/conn.py", line 586, in recv
2016-09-28 09:19:07.860 29051 ERROR kafka.producer return self._recv()
2016-09-28 09:19:07.860 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/conn.py", line 593, in _recv
2016-09-28 09:19:07.860 29051 ERROR kafka.producer data = self._sock.recv(bytes_to_read)
2016-09-28 09:19:07.860 29051 ERROR kafka.producer AttributeError: 'NoneType' object has no attribute 'recv'
------------------------------------------------------------------------------------------------------------------------------------

2016-09-28 09:19:07.926 29051 ERROR kafka.producer [req-b9f7c456-5d38-4d9c-8869-a1aeb9409879 - - - - -] Unable to send messages
2016-09-28 09:19:07.926 29051 ERROR kafka.producer Traceback (most recent call last):
2016-09-28 09:19:07.926 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py", line 422, in _send_messages
2016-09-28 09:19:07.926 29051 ERROR kafka.producer fail_on_error=self.sync_fail_on_error
2016-09-28 09:19:07.926 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 644, in send_produce_request
2016-09-28 09:19:07.926 29051 ERROR kafka.producer resps = self._send_broker_aware_request(payloads, encoder, decoder)
2016-09-28 09:19:07.926 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 286, in _send_broker_aware_request
2016-09-28 09:19:07.926 29051 ERROR kafka.producer conn.recv()
2016-09-28 09:19:07.926 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/conn.py", line 565, in recv
2016-09-28 09:19:07.926 29051 ERROR kafka.producer assert not self._processing, 'Recursion not supported'
2016-09-28 09:19:07.926 29051 ERROR kafka.producer AssertionError: Recursion not supported
2016-09-28 09:19:07.926 29051 ERROR kafka.producer
----------------------------------------------------------------------------------------------------------------------------------------
2016-09-28 09:19:07.931 29051 ERROR kafka.producer [req-17551213-3624-4aee-a083-b22be5e9c0e5 - - - - -] Unable to send messages
2016-09-28 09:19:07.931 29051 ERROR kafka.producer Traceback (most recent call last):
2016-09-28 09:19:07.931 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py", line 422, in _send_messages
2016-09-28 09:19:07.931 29051 ERROR kafka.producer fail_on_error=self.sync_fail_on_error
2016-09-28 09:19:07.931 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 648, in send_produce_request
2016-09-28 09:19:07.931 29051 ERROR kafka.producer (not fail_on_error or not self._raise_on_response_error(resp))]
2016-09-28 09:19:07.931 29051 ERROR kafka.producer raise resp
2016-09-28 09:19:07.931 29051 ERROR kafka.producer FailedPayloadsError: FailedPayloadsError

---------------------------------------------------------------------------------------------------------------------------------------
2016-09-28 09:19:07.931 29051 ERROR kafka.producer
2016-09-28 09:19:07.933 29051 ERROR kafka.producer [req-70bccdbc-e37a-43d3-9cdd-ba663db4f338 - - - - -] Unable to send messages
2016-09-28 09:19:07.933 29051 ERROR kafka.producer Traceback (most recent call last):
2016-09-28 09:19:07.933 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/producer/base.py", line 422, in _send_messages
2016-09-28 09:19:07.933 29051 ERROR kafka.producer fail_on_error=self.sync_fail_on_error
2016-09-28 09:19:07.933 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 644, in send_produce_request
2016-09-28 09:19:07.933 29051 ERROR kafka.producer resps = self._send_broker_aware_request(payloads, encoder, decoder)
2016-09-28 09:19:07.933 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 264, in _send_broker_aware_request
2016-09-28 09:19:07.933 29051 ERROR kafka.producer future = conn.send(request, expect_response=expect_response)
2016-09-28 09:19:07.933 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/conn.py", line 512, in send
2016-09-28 09:19:07.933 29051 ERROR kafka.producer return self._send(request, expect_response=expect_response)
2016-09-28 09:19:07.933 29051 ERROR kafka.producer File "/usr/local/lib/python2.7/dist-packages/kafka/conn.py", line 535, in _send
2016-09-28 09:19:07.933 29051 ERROR kafka.producer self._sock.setblocking(False)
2016-09-28 09:19:07.933 29051 ERROR kafka.producer AttributeError: 'NoneType' object has no attribute 'setblocking'
-------------------------------------------------------------------------------------------------------------------------------------

Is ceilometer kafka publisher is stable one ?

Revision history for this message
raghunath (raghunath-d) wrote :
Revision history for this message
gordon chung (chungg) wrote :

i don't use it. you need to ask cisco/fujitsu folks who implemented it probably.

Revision history for this message
celik esra (celik-esra) wrote :

I faced the same error and finally made it work by changing the deprecated SimpleProducer method to KafkaProducer at kafka_broker.py. I will soon send a patch for this.

Changed in ceilometer:
assignee: nobody → celik esra (celik-esra)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/392637

Changed in ceilometer:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/392637
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=e0a28535b91e81f2b8d4b6c405bf35e7651d1000
Submitter: Jenkins
Branch: master

commit e0a28535b91e81f2b8d4b6c405bf35e7651d1000
Author: celik.esra <email address hidden>
Date: Wed Nov 2 15:47:36 2016 +0300

    Replace SimpleProducer with KafkaProducer

    SimpleProducer is marked as deprecated at kafka-python library [1].
    The deprecated method causes "FailedPayloadsError: FailedPayloadsError" error.
    This patch replaces SimpleProducer with KafkaProducer.

    [1] http://kafka-python.readthedocs.io/en/1.1.1/simple.html

    Change-Id: I7c979e1064c9c42e82ae71e2631a3d2bd0462839
    Closes-bug: #1628456

Changed in ceilometer:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 8.0.0

This issue was fixed in the openstack/ceilometer 8.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.