rabbitMQ locks all services for 10 secs after creating instance snapshot

[Diogo Monteiro]

Steps to reproduce the problem.
#1 Create a private template from a running virtual instance
#2 Create a new virtual instance using the private template

Result:
All openstack services lock for about 10 secs -- api server stops responding to any requests.

The problem is always reproducible.

Error logs:

1. Create a private template from a running virtual instance
2. Create a new virtual instance using the private template

Observations:

RabbitMQ Logs:

=WARNING REPORT==== 15-Aug-2014::15:29:46 ===
exception on TCP connection <0.13976.3> from 127.0.0.1:52700
connection_closed_abruptly

=INFO REPORT==== 15-Aug-2014::15:29:46 ===
closing TCP connection <0.13976.3> from 127.0.0.1:52700

=WARNING REPORT==== 15-Aug-2014::15:29:46 ===
exception on TCP connection <0.13980.3> from 127.0.0.1:52701
connection_closed_abruptly

=INFO REPORT==== 15-Aug-2014::15:29:46 ===
closing TCP connection <0.13980.3> from 127.0.0.1:52701

In nova-conductor

2014-08-15 09:30:03.677 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.119076 sec
2014-08-15 09:30:14.506 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.823113 sec
2014-08-15 09:33:59.423 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.470631 sec
2014-08-15 14:01:12.217 30924 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 2.697045 sec
2014-08-15 14:06:10.072 30924 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.716281 sec
2014-08-15 14:33:08.902 51990 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 2.419363 sec

Overview of the environment:
OS: 3.8.0-35-generic #50~precise1-Ubuntu SMP Wed Dec 4 17:25:51 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Openstack version: havana

Services:
nova - 2.15.0
neutron - 2.3.0
glance - 0.11.0
cinder - 1.0.6
keystone - 0.3.2

RabbitMQ
 {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                        {os_mon,"CPO CXC 138 46","2.2.7"},
                        {sasl,"SASL CXC 138 11","2.1.10"},
                        {mnesia,"MNESIA CXC 138 12","4.5"},
                        {stdlib,"ERTS CXC 138 10","1.17.5"},
                        {kernel,"ERTS CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,30703760},
          {processes,14640352},
          {processes_used,14630712},
          {system,16063408},
          {atom,1124441},
          {atom_used,1120308},
          {binary,478496},
          {code,11134417},
          {ets,986440}]},
 {vm_memory_high_watermark,0.39999999997532165},
 {vm_memory_limit,3241710387}]