| 2014-04-07 19:09:30 |
Russell Bryant |
description |
The following exception was originally observed against the old rpc code, but the same problem exists in oslo.messaging.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
return infunc(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
self.consume()
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
it.next()
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
yield self.ensure(_error_callback, _consume)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
return method(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
nxt_receiver = self.session.next_receiver(timeout=timeout)
File "<string>", line 6, in next_receiver
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
if self._ecwait(lambda: self.incoming, timeout):
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
result = self._ewait(lambda: self.closed or predicate(), timeout)
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
result = self.connection._ewait(lambda: self.error or predicate(), timeout)
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
self.check_error()
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
raise self.error
InternalError: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 667, in write
self._op_dec.write(*self._seg_dec.read())
File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
if self.op.headers is None:
AttributeError: 'NoneType' object has no attribute 'headers'
It's possible for something to put the qpid client into a bad state. In particular, I have observed a case that will cause session.next_receiver() to immediately raise an AttributeError. This exception makes it all the way out. If the eventlet executor is used, the forever_retry_uncaught_exceptions() decorator will get hit. It will go back into this code and get the same error, stuck in an infinite loop of retrying.
The connection needs to be reset in this case to recover. |
The following exception was originally observed against the old rpc code, but the same problem exists in oslo.messaging.
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
return infunc(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
self.consume()
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
it.next()
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
yield self.ensure(_error_callback, _consume)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
return method(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
nxt_receiver = self.session.next_receiver(timeout=timeout)
File "<string>", line 6, in next_receiver
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 665, in next_receiver
if self._ecwait(lambda: self.incoming, timeout):
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
result = self._ewait(lambda: self.closed or predicate(), timeout)
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 571, in _ewait
result = self.connection._ewait(lambda: self.error or predicate(), timeout)
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 214, in _ewait
self.check_error()
File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 207, in check_error
raise self.error
InternalError: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 667, in write
self._op_dec.write(*self._seg_dec.read())
File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
if self.op.headers is None:
AttributeError: 'NoneType' object has no attribute 'headers'
It's possible for something to put the qpid client into a bad state. In particular, I have observed a case that will cause session.next_receiver() to immediately raise an InternalError. This exception makes it all the way out. If the eventlet executor is used, the forever_retry_uncaught_exceptions() decorator will get hit. It will go back into this code and get the same error, stuck in an infinite loop of retrying.
The connection needs to be reset in this case to recover. |
|
