When I run "service rabbitmq-server stop" on one node to simulating a failure, I got following error logs, and the consumer can't failover from the bad node. It will reconnect the failure node forever instead of other nodes. "kombu_failover_strategy" is default value of "round-robin".
2009-01-13 18:32:42.785 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [4e976d46-ceee-4617-b9be-5e4821990738] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed
2009-01-13 18:32:43.819 17 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed
2009-01-13 18:32:43.819 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...
2009-01-13 18:32:58.874 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [4e976d46-ceee-4617-b9be-5e4821990738] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed
2009-01-13 18:32:59.907 17 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed
2009-01-13 18:32:59.907 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...
I have a rabbitmq cluster of 3 nodes
root@47704165d2 bb:/# rabbitmqctl cluster_status [{disc, [rabbit@ 0482398a286e, rabbit@ 3709521b608a,
rabbit@ 47704165d2bb] }]}, nodes,[ rabbit@ 0482398a286e, rabbit@ 3709521b608a, rabbit@ 47704165d2bb] }, name,<< "rabbit@ 47704165d2bb" >>}, [{rabbit@ 0482398a286e, []},
{rabbit@ 3709521b608a, []},
{rabbit@ 47704165d2bb, []}]}] bb:/# rabbitmqctl list_policies
Cluster status of node rabbit@47704165d2bb ...
[{nodes,
{running_
{cluster_
{partitions,[]},
{alarms,
root@47704165d2
Listing policies ...
/ ha-all all ^ha\\. {"ha-mode":"all"} 0
My oslo_message client configuration _rabbit] hosts=120. 0.0.56: 5671,120. 0.0.57: 5671,120. 0.0.55: 5671 password= cloud ha_queues= True retry_interval= 1 retry_backoff= 2 max_retries= 0 durable_ queues= False
[oslo_messaging
rabbit_
rabbit_userid=cloud
rabbit_
rabbit_
rabbit_
rabbit_
rabbit_
rabbit_
When I run "service rabbitmq-server stop" on one node to simulating a failure, I got following error logs, and the consumer can't failover from the bad node. It will reconnect the failure node forever instead of other nodes. "kombu_ failover_ strategy" is default value of "round-robin".
2009-01-13 18:32:42.785 17 ERROR oslo.messaging. _drivers. impl_rabbit [-] [4e976d46- ceee-4617- b9be-5e48219907 38] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed _drivers. impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed _drivers. impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying... _drivers. impl_rabbit [-] [4e976d46- ceee-4617- b9be-5e48219907 38] AMQP server 120.0.0.56:5671 closed the connection. Check login credentials: Socket closed _drivers. impl_rabbit [-] Unable to connect to AMQP server on 120.0.0.56:5671 after None tries: Socket closed _drivers. impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...
2009-01-13 18:32:43.819 17 ERROR oslo.messaging.
2009-01-13 18:32:43.819 17 WARNING oslo.messaging.
2009-01-13 18:32:58.874 17 ERROR oslo.messaging.
2009-01-13 18:32:59.907 17 ERROR oslo.messaging.
2009-01-13 18:32:59.907 17 WARNING oslo.messaging.
Who can help me. Thanks!