Seeing similar messages in the following services. g-api, n-api, q-svc, n-cond, n-sch, n-cpu, c-sch, c-vol
Setup: devstack(Master)+Ubunut14.04
Outpu for c-vol:
2015-04-14 13:11:33.153 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.122.95:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. 2015-04-14 13:11:34.167 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.122.95:5672 2015-04-14 13:11:36.169 DEBUG oslo_messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: from (pid=30954) on_error /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:782 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit Traceback (most recent call last): 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 440, in _ensured 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 512, in __call__ 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 831, in execute_method 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit method() 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1024, in _consume 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit rate=self.driver_conf.heartbeat_rate) 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 268, in heartbeat_check 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit return self.transport.heartbeat_check(self.connection, rate=rate) 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 127, in heartbeat_check 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit return connection.heartbeat_tick(rate=rate) 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 876, in heartbeat_tick 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit raise ConnectionForced('Too many heartbeats missed') 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit ConnectionForced: Too many heartbeats missed 2015-04-14 13:11:36.169 TRACE oslo_messaging._drivers.impl_rabbit 2015-04-14 13:11:36.169 ERROR oslo_messaging._drivers.impl_rabbit [-] AMQP server on 192.168.122.95:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. 2015-04-14 13:11:37.187 INFO oslo_messaging._drivers.impl_rabbit [-] Reconnected to AMQP server on 192.168.122.95:5672
Seeing similar messages in the following services.
g-api, n-api, q-svc, n-cond, n-sch, n-cpu, c-sch, c-vol
Setup: devstack( Master) +Ubunut14. 04
Outpu for c-vol:
2015-04-14 13:11:33.153 ERROR oslo_messaging. _drivers. impl_rabbit [-] AMQP server on 192.168.122.95:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. _drivers. impl_rabbit [-] Reconnected to AMQP server on 192.168.122.95:5672 _drivers. impl_rabbit [-] Received recoverable error from kombu: from (pid=30954) on_error /usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py:782 _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 440, in _ensured _drivers. impl_rabbit return fun(*args, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 512, in __call__ _drivers. impl_rabbit return fun(*args, channel= channels[ 0], **kwargs), channels[0] _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py", line 831, in execute_method _drivers. impl_rabbit method() _drivers. impl_rabbit File "/usr/local/ lib/python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py", line 1024, in _consume _drivers. impl_rabbit rate=self. driver_ conf.heartbeat_ rate) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 268, in heartbeat_check _drivers. impl_rabbit return self.transport. heartbeat_ check(self. connection, rate=rate) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqp. py", line 127, in heartbeat_check _drivers. impl_rabbit return connection. heartbeat_ tick(rate= rate) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 876, in heartbeat_tick _drivers. impl_rabbit raise ConnectionForce d('Too many heartbeats missed') _drivers. impl_rabbit ConnectionForced: Too many heartbeats missed _drivers. impl_rabbit _drivers. impl_rabbit [-] AMQP server on 192.168.122.95:5672 is unreachable: Too many heartbeats missed. Trying again in 1 seconds. _drivers. impl_rabbit [-] Reconnected to AMQP server on 192.168.122.95:5672
2015-04-14 13:11:34.167 INFO oslo_messaging.
2015-04-14 13:11:36.169 DEBUG oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 TRACE oslo_messaging.
2015-04-14 13:11:36.169 ERROR oslo_messaging.
2015-04-14 13:11:37.187 INFO oslo_messaging.