Seeing the following stack trace on solum-worker screen:
load_log /opt/stack/solum/solum/uploaders/local.py:26 2015-08-04 18:51:04.573 31699 DEBUG oslo_messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:627 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__ 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 674, in execute_method 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit method() 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 900, in _consume 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout) 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit chanmap, None, timeout=timeout, 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout) 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.method_reader.read_method() 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit raise m 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit IOError: Socket closed 2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit 2015-08-04 18:51:04.575 31699 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 10.0.2.15:5672 closed the connection. Check login credentials: Socket closed 2015-08-04 18:51:05.590 31699 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 10.0.2.15:5672
Seeing the following stack trace on solum-worker screen:
load_log /opt/stack/ solum/solum/ uploaders/ local.py: 26 _drivers. impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py:627 _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ kombu/connectio n.py", line 436, in _ensured _drivers. impl_rabbit return fun(*args, **kwargs) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ kombu/connectio n.py", line 508, in __call__ _drivers. impl_rabbit return fun(*args, channel= channels[ 0], **kwargs), channels[0] _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py", line 674, in execute_method _drivers. impl_rabbit method() _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py", line 900, in _consume _drivers. impl_rabbit self.connection .drain_ events( timeout= poll_timeout) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ kombu/connectio n.py", line 275, in drain_events _drivers. impl_rabbit return self.transport. drain_events( self.connection , **kwargs) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ kombu/transport /pyamqp. py", line 95, in drain_events _drivers. impl_rabbit return connection. drain_events( **kwargs) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ amqp/connection .py", line 302, in drain_events _drivers. impl_rabbit chanmap, None, timeout=timeout, _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ amqp/connection .py", line 365, in _wait_multiple _drivers. impl_rabbit channel, method_sig, args, content = read_timeout( timeout) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ amqp/connection .py", line 336, in read_timeout _drivers. impl_rabbit return self.method_ reader. read_method( ) _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ amqp/method_ framing. py", line 189, in read_method _drivers. impl_rabbit raise m _drivers. impl_rabbit IOError: Socket closed _drivers. impl_rabbit _drivers. impl_rabbit [-] AMQP server 10.0.2.15:5672 closed the connection. Check login credentials: Socket closed _drivers. impl_rabbit [-] Reconnected to AMQP server on 10.0.2.15:5672
2015-08-04 18:51:04.573 31699 DEBUG oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging.
2015-08-04 18:51:04.575 31699 ERROR oslo_messaging.
2015-08-04 18:51:05.590 31699 INFO oslo_messaging.