Cannot publish errors to ceilometer

[Li Ma]

I tried to test sending 'notification-errors' to ceilometer to report ERROR log in neutron, but I failed with RuntimeError:

RuntimeError: maximum recursion depth exceeded

It seems that the ERROR message causes infinite loop on notifying to ceilometer.

The traceback is linked at: http://paste.openstack.org/show/87907/

[Eugene Nikanorov (enikanorov)]

I see python2.6 in the trace. Is the problem specific to py26? What operation system are you using and what is neutron version?

[Li Ma (nick-ma-z)]

Distro: RHEL 6.5
Python: 2.6.6
Neutron: icehouse stable release

I checked the git master before, I think the code are the same. I'm not sure whether it is related with python 2.6, I'll try to find other environments.

This can be easily reproduced by:

setting publish_errors = True in neutron.conf

after you make notification work well.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[David Hill (david-hill-ubisoft)]

We have a similar issue here but without Neutron...

[David Hill (david-hill-ubisoft)]

We're using :
openstack-ceilometer-compute-2014.1.3

[gordon chung (chungg)]

based on the paste this doesn't seem to be a ceilometer issue. i've added oslo.messaging.

[Doug Hellmann (doug-hellmann)]

It looks like this might be an interaction between the RPC and logging notifiers. Which notifier plugins do you have active?

[gordon chung (chungg)]

marking ceilometer part invalid as based on logs, it never reaches queue. ceilometer currently listens to error priority as of kilo

[Sachin (smanpathak)]

I saw this in the context of cinder volume service. Basically, if one sets following options in the openstack service config (cinder.conf in this example)
notification_driver=messaging
publish_errors=True

After this, if the rpc connection breaks, the oslo.messaging code invokes LOG.error, which in turn calls oslo.rpc and recursion ensues.
The worst part is that all green threads eventually lock up. The service cannot continue even though rpc connection is restored.

[Launchpad Janitor (janitor)]

[Expired for oslo.messaging because there has been no activity for 60 days.]