Comment 38 for bug 1371723

Revision history for this message
Sean Murphy (murp) wrote : Re: To many reconnects in logs

We have MOS 5.1.1 installed but we still see some behaviour as described here. Here are the relevant packages on our controller:

root@node-1:~# dpkg -l | grep nova-api
ii nova-api 1:2014.1.3-fuel5.1.1~mira3 OpenStack Compute - API frontend

root@node-1:~# dpkg -l | grep oslo
ii python-oslo.config 1:1.2.1-0ubuntu1~cloud0 Common code for Openstack Projects (configuration API)
ii python-oslo.i18n 1.0.0-fuel5.1.1~mira1 Oslo Internationalization Utilities - Python 2.x
ii python-oslo.messaging 1.3.1-fuel5.1.1~mira0 oslo messaging library
ii python-oslo.rootwrap 1.0.0-0ubuntu2 Fine filtering of shell commands as 'root'
ii python-oslo.serialization 1.0.0-fuel5.1.1~mira0 utilities for serialization , especially JSON - Python 2.x
ii python-oslo.utils 1.0.0-fuel5.1.1~mira0 set of utility functions for OpenStack - Python 2.x
ii python-oslo.vmware 0.2-0ubuntu1 oslo vmware library

root@node-1:~# dpkg -l | grep rabbit
ii rabbitmq-server 3.3.5-ubuntu3 AMQP server written in Erlang

We see issues that the rabbit closes connections to many of the processes (it seems many of the
nova-api processes); when there is new request to nova-api, it often has a broken pipe and then
it started a new connection to rabbitmq. So we have this sequence:

- rabbit kills connection to many nova-api processes due to heartbeat timeouts
- nova-api receives request
- nova-api tries to talk to rabbit but experiences broken pipe
- (many) nova-api processes restart communications with rabbit
- nova-api is able to then communicate

It does not have much of an impact in userland, but it's suboptimal from pov of keeping
clean logs.