Comment 0 for bug 1362155

{"build_id": "2014-07-17_11-18-10", "mirantis": "yes", "build_number": "135", "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f", "nailgun_sha": "1d08d6f80b6514085dd8c0af4d437ef5d37e2802", "production": "docker", "api": "1.0", "fuelmain_sha": "c8e13df4c7de3ce3504c2bcb6d51a165b9aae0b6", "astute_sha": "9a74b788be9a7c5682f1c52a892df36e4766ce3f", "release": "5.0.1", "fuellib_sha": "e8c2bb726be6b78c3a34f75c84337a3a5662bb35"}

RabbitMQ log contains constantly appearing error messages like shown below:

================================================================================
=INFO REPORT==== 27-Aug-2014::11:43:11 ===
accepting AMQP connection <0.27627.0> (192.168.0.2:54186 -> 192.168.0.2:5672)

=INFO REPORT==== 27-Aug-2014::11:43:17 ===
accepting AMQP connection <0.27743.0> (192.168.0.2:54188 -> 192.168.0.2:5672)

=ERROR REPORT==== 27-Aug-2014::11:43:21 ===
closing AMQP connection <0.27627.0> (192.168.0.2:54186 -> 192.168.0.2:5672):
{handshake_timeout,handshake}

=ERROR REPORT==== 27-Aug-2014::11:43:27 ===
closing AMQP connection <0.27743.0> (192.168.0.2:54188 -> 192.168.0.2:5672):
{handshake_timeout,handshake}

=INFO REPORT==== 27-Aug-2014::11:43:52 ===
accepting AMQP connection <0.27918.0> (192.168.0.2:54189 -> 192.168.0.2:5672)

=ERROR REPORT==== 27-Aug-2014::11:44:02 ===
closing AMQP connection <0.27918.0> (192.168.0.2:54189 -> 192.168.0.2:5672):
{handshake_timeout,handshake}
================================================================================

In the same time there are trace messages in OpenStack logs, for example, /var/log/nova-all.log

================================================================================
<182>Aug 27 10:41:38 node-1 nova-urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.0.2
<182>Aug 27 10:41:38 node-1 nova-nova.osapi_compute.wsgi.server INFO: 172.16.40.136 "GET /v2/631fa1bc73f84530b78c3efcfa38ebfd/servers/0a64f095-58d3-43f4-b81d-39e0864fa0b7 HTTP/1.1" status: 200 len: 2038 time: 0.1334660
<182>Aug 27 10:41:39 node-1 nova-nova.osapi_compute.wsgi.server INFO: 172.16.40.136 "GET /v2/631fa1bc73f84530b78c3efcfa38ebfd/servers/0a64f095-58d3-43f4-b81d-39e0864fa0b7/os-volume_attachments HTTP/1.1" status: 200 len: 197 time: 0.0401199
<182>Aug 27 10:41:39 node-1 nova-nova.osapi_compute.wsgi.server INFO: 172.16.40.136 "GET /v2/631fa1bc73f84530b78c3efcfa38ebfd/flavors/2 HTTP/1.1" status: 200 len: 600 time: 0.0119381
<179>Aug 27 10:41:39 node-1 nova-oslo.messaging._drivers.impl_rabbit ERROR: Failed to publish message to topic 'compute.node-3.vm.mirantis.net': [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 632, in ensure
    return method(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 728, in _publish
    publisher = cls(self.conf, self.channel, topic, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 384, in __init__
    **options)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 331, in __init__
    self.reconnect(channel)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in reconnect
    routing_key=self.routing_key)
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 84, in __init__
    self.revive(self._channel)
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 218, in revive
    self.declare()
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 104, in declare
    self.exchange.declare()
  File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 166, in declare
    nowait=nowait, passive=passive,
  File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 613, in exchange_declare
    self._send_method((40, 10), args)
  File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 56, in _send_method
    self.channel_id, method_sig, args, content,
  File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 221, in write_method
    write_frame(1, channel, payload)
  File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 177, in write_frame
    frame_type, channel, size, payload, 0xce,
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 309, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer
_send_method((40, 10), args)
  File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 56, in _send_method
    self.channel_id, method_sig, args, content,
  File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 221, in write_method
    write_frame(1, channel, payload)
  File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 177, in write_frame
    frame_type, channel, size, payload, 0xce,
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 309, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer
<182>Aug 27 10:41:39 node-1 nova-oslo.messaging._drivers.impl_rabbit INFO: Reconnecting to AMQP server on 192.168.0.2:5672
<182>Aug 27 10:41:39 node-1 nova-oslo.messaging._drivers.impl_rabbit INFO: Delaying reconnect for 5.0 seconds...
<182>Aug 27 10:41:44 node-1 nova-oslo.messaging._drivers.impl_rabbit INFO: Connected to AMQP server on 192.168.0.2:5672
<180>Aug 27 10:41:44 node-1 nova-nova.consoleauth.manager AUDIT: Received Token: 14eac98c-280b-40ae-aee0-2a658cf25060, {'instance_uuid': u'0a64f095-58d3-43f4-b81d-39e0864fa0b7', 'internal_access_path': None, 'last_activity_at': 1409136104.2732379, 'console_type': u'novnc', 'host': u'192.168.0.4', 'token': u'14eac98c-280b-40ae-aee0-2a658cf25060', 'port': u'5903'}
<179>Aug 27 10:41:44 node-1 nova-oslo.messaging._drivers.impl_rabbit ERROR: Failed to publish message to topic 'reply_96632cd2e06642d4ad0d860256d55c66': [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 632, in ensure
    return method(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 728, in _publish
    publisher = cls(self.conf, self.channel, topic, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 365, in __init__
    type='direct', **options)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 331, in __init__
    self.reconnect(channel)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in reconnect
    routing_key=self.routing_key)
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 84, in __init__
    self.revive(self._channel)
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 218, in revive
    self.declare()
  File "/usr/lib/python2.6/site-packages/kombu/messaging.py", line 104, in declare
    self.exchange.declare()
  File "/usr/lib/python2.6/site-packages/kombu/entity.py", line 166, in declare
    nowait=nowait, passive=passive,
  File "/usr/lib/python2.6/site-packages/amqp/channel.py", line 613, in exchange_declare
    self._send_method((40, 10), args)
  File "/usr/lib/python2.6/site-packages/amqp/abstract_channel.py", line 56, in _send_method
    self.channel_id, method_sig, args, content,
  File "/usr/lib/python2.6/site-packages/amqp/method_framing.py", line 221, in write_method
    write_frame(1, channel, payload)
  File "/usr/lib/python2.6/site-packages/amqp/transport.py", line 177, in write_frame
    frame_type, channel, size, payload, 0xce,
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 309, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 295, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer
<182>Aug 27 10:41:44 node-1 nova-oslo.messaging._drivers.impl_rabbit INFO: Reconnecting to AMQP server on 192.168.0.2:5672
<182>Aug 27 10:41:44 node-1 nova-oslo.messaging._drivers.impl_rabbit INFO: Delaying reconnect for 5.0 seconds...
================================================================================

Looks like the bug https://bugs.launchpad.net/ceilometer/+bug/1337715