This is new to me, and I'm not sure if the logging change is causing it:
http://logs.openstack.org/31/124231/3/gate/gate-tempest-dsvm-full/b860d61/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-09-27_14_47_26_507
2014-09-27 14:47:26.507 ERROR oslo.messaging.rpc.dispatcher [req-e79201b2-ad39-4f1e-9d45-7c6daf90af52 TestEncryptedCinderVolumes-703905565 TestEncryptedCinderVolumes-2029807785] Exception during message handling: <type 'NoneType'> can't be decoded 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last): 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher incoming.message)) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 414, in decorated_function 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 88, in wrapped 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher payload) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 71, in wrapped 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 298, in decorated_function 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher pass 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 284, in decorated_function 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 326, in decorated_function 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__ 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 314, in decorated_function 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 4619, in detach_volume 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher self._detach_volume(context, instance, bdm) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 4565, in _detach_volume 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher connection_info = jsonutils.loads(bdm.connection_info) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/jsonutils.py", line 188, in loads 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher return json.loads(strutils.safe_decode(s, encoding), **kwargs) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/openstack/common/strutils.py", line 145, in safe_decode 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher raise TypeError("%s can't be decoded" % type(text)) 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher TypeError: <type 'NoneType'> can't be decoded 2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.rpc.dispatcher
I guess it's not new (so not related to my logging change):
http://goo.gl/VDYRk3
That goes back before 9/27.
This is new to me, and I'm not sure if the logging change is causing it:
http:// logs.openstack. org/31/ 124231/ 3/gate/ gate-tempest- dsvm-full/ b860d61/ logs/screen- n-cpu.txt. gz?level= TRACE#_ 2014-09- 27_14_47_ 26_507
2014-09-27 14:47:26.507 ERROR oslo.messaging. rpc.dispatcher [req-e79201b2- ad39-4f1e- 9d45-7c6daf90af 52 TestEncryptedCi nderVolumes- 703905565 TestEncryptedCi nderVolumes- 2029807785] Exception during message handling: <type 'NoneType'> can't be decoded rpc.dispatcher Traceback (most recent call last): rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 134, in _dispatch_and_reply rpc.dispatcher incoming.message)) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 177, in _dispatch rpc.dispatcher return self._do_ dispatch( endpoint, method, ctxt, args) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 123, in _do_dispatch rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 414, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ new/nova/ nova/exception. py", line 88, in wrapped rpc.dispatcher payload) rpc.dispatcher File "/opt/stack/ new/nova/ nova/openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ new/nova/ nova/exception. py", line 71, in wrapped rpc.dispatcher return f(self, context, *args, **kw) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 298, in decorated_function rpc.dispatcher pass rpc.dispatcher File "/opt/stack/ new/nova/ nova/openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 284, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 326, in decorated_function rpc.dispatcher kwargs['instance'], e, sys.exc_info()) rpc.dispatcher File "/opt/stack/ new/nova/ nova/openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 314, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 4619, in detach_volume rpc.dispatcher self._detach_ volume( context, instance, bdm) rpc.dispatcher File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 4565, in _detach_volume rpc.dispatcher connection_info = jsonutils. loads(bdm. connection_ info) rpc.dispatcher File "/opt/stack/ new/nova/ nova/openstack/ common/ jsonutils. py", line 188, in loads rpc.dispatcher return json.loads( strutils. safe_decode( s, encoding), **kwargs) rpc.dispatcher File "/opt/stack/ new/nova/ nova/openstack/ common/ strutils. py", line 145, in safe_decode rpc.dispatcher raise TypeError("%s can't be decoded" % type(text)) rpc.dispatcher TypeError: <type 'NoneType'> can't be decoded rpc.dispatcher
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
2014-09-27 14:47:26.507 25061 TRACE oslo.messaging.
I guess it's not new (so not related to my logging change):
http:// goo.gl/ VDYRk3
That goes back before 9/27.