Error in rpcpublisher

[Mehdi Abaakouk]

Hi,

I have sometimes this following backtrace:

2013-08-12 15:34:14.752 29702 ERROR ceilometer.pipeline [-] Pipeline cpu_pipeline: Continue after error from publisher <ceilometer.publisher.rpc.RPCPublisher object at 0x39cee90>
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline Traceback (most recent call last):
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline File "/vagrant/stack-master/ceilometer/ceilometer/pipeline.py", line 214, in _publish_counters
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline p.publish_counters(ctxt, transformed_counters)
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline File "/vagrant/stack-master/ceilometer/ceilometer/publisher/rpc.py", line 178, in publish_counters
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline self.flush()
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline File "/vagrant/stack-master/ceilometer/ceilometer/publisher/rpc.py", line 224, in flush
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline self.local_queue.pop(0)
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline IndexError: pop from empty list
2013-08-12 15:34:14.752 29702 TRACE ceilometer.pipeline

Cheers,

[gordon chung (chungg)]

i see this as well... do you think your patch to control publisher calls (https://review.openstack.org/#/c/39510/) will resolve this? seems to work for me but this could be just hiding the real bug.

[Mehdi Abaakouk (sileht)]

No this patch doesn't fix this issue, when I have written the code I have think that publish_counters cannot be called in a concurrent manner. And the code of the RPCPublisher.flush method is not ThreadSafe

But (I haven't checked yet), I think that gevent can stop on the rpc.cast call (because it does some IO), to handle a other publish_counters call. And because RPCPublisher.flush method is not thread safe, self.local_queue can be override by the seconds publish_counters call, and when the gevent choose to finish to run the first call the local_queue doesn't have anymore the excepted content.

I put this issue as Critical because in this case some samples are loose.

[Mehdi Abaakouk (sileht)]

s/gevent/eventlet/g :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/42587

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/42587
Committed: http://github.com/openstack/ceilometer/commit/e7e74f7dc311fa2695514ab32584f849aecc6a1d
Submitter: Jenkins
Branch: master

commit e7e74f7dc311fa2695514ab32584f849aecc6a1d
Author: Mehdi Abaakouk <email address hidden>
Date: Mon Aug 19 10:48:18 2013 +0200

    Make RPCPublisher flush method threadsafe

    This change allow concurrent access to the local queue of the
    RPCPublisher

    Fixes bug #1211736

    Change-Id: I13371329d40e43f42c357a0893e4023f343a5efa