Rabbitmq error in solum-worker

Bug #1481457 reported by Devdatta Kulkarni
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Solum
New
Undecided
Unassigned

Bug Description

Seeing the following stack trace on solum-worker screen.

Solum commit: 18a33a5bd4470c722894ee9a2abf97f2441bd52b
Devstack commit: 57036ba2b77eb1f09449266fd7510cddfc9ec321
kombu: 3.0.26
oslo.messaging: 2.2.0
rabbitmq-server: Version: 3.2.4-1

load_log /opt/stack/solum/solum/uploaders/local.py:26
2015-08-04 18:51:04.573 31699 DEBUG oslo_messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:627
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 674, in execute_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 900, in _consume
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit return self.method_reader.read_method()
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit raise m
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit IOError: Socket closed
2015-08-04 18:51:04.573 31699 TRACE oslo_messaging._drivers.impl_rabbit
2015-08-04 18:51:04.575 31699 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server 10.0.2.15:5672 closed the connection. Check login credentials: Socket closed
2015-08-04 18:51:05.590 31699 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 10.0.2.15:5672

Tags: solum-infra
description: updated
tags: added: solum-infra
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.