| 2015-08-04 19:15:29 |
Devdatta Kulkarni |
description |
Seeing the following stack trace on solum-worker screen:
load_log /opt/stack/solum/solum/uploaders/local.py:26
2015-08-04 18:51:04.573 31699 DEBUG oslo_messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:627
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 674, in execute_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 900, in _consume
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.method_reader.read_method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit raise m
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit IOError: Socket closed
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit
2015-08-04 18:51:04.575 31699 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 10.0.2.15:5672 closed the connection. Check login credentials: Socket closed
2015-08-04 18:51:05.590 31699 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 10.0.2.15:5672 |
Seeing the following stack trace on solum-worker screen.
Solum commit: 18a33a5bd4470c722894ee9a2abf97f2441bd52b
Devstack commit: 57036ba2b77eb1f09449266fd7510cddfc9ec321
kombu: 3.0.26
oslo.messaging: 2.2.0
rabbitmq-server: Version: 3.2.4-1
load_log /opt/stack/solum/solum/uploaders/local.py:26
2015-08-04 18:51:04.573 31699 DEBUG oslo_messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:627
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 674, in execute_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 900, in _consume
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.method_reader.read_method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit raise m
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit IOError: Socket closed
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit
2015-08-04 18:51:04.575 31699 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 10.0.2.15:5672 closed the connection. Check login credentials: Socket closed
2015-08-04 18:51:05.590 31699 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 10.0.2.15:5672 |
|
