aodh listener does not retry to connect to rpc if connection failure
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Aodh |
Invalid
|
Medium
|
Liusheng |
Bug Description
If Aodh Listener can't connect to AMQP, it won't loop to try again later, like it's done in many other OpenStack services, and the process will fail.
See:
Aodh Listener tries to start:
http://
But RabbitMQ resources for Aodh are created after:
http://
Aodh Listener fails to start:
http://
And never try again. That's a bug because other OpenStack services use to loop again.
Changed in aodh: | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in aodh: | |
assignee: | nobody → Liusheng (liusheng) |
milestone: | none → newton-1 |
I have tried to reproduce this issue, after I stop rabbitmq-server in my devstack and then I try to start aodh-notifier, it will try again to connect rabbitmq, see:
2016-04-07 11:28:19.743 98183 DEBUG oslo_service. service [-] coordination. backend_ url = None log_opt_values /usr/local/ lib/python2. 7/dist- packages/ oslo_config/ cfg.py: 2525 service [-] coordination. check_watchers = 10.0 log_opt_values /usr/local/ lib/python2. 7/dist- packages/ oslo_config/ cfg.py: 2525 service [-] coordination. heartbeat = 1.0 log_opt_values /usr/local/ lib/python2. 7/dist- packages/ oslo_config/ cfg.py: 2525 service [-] ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* ******* *** log_opt_values /usr/local/ lib/python2. 7/dist- packages/ oslo_config/ cfg.py: 2527 _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. _drivers. impl_rabbit [-] AMQP server on 10.229.40.107:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds.
2016-04-07 11:28:19.743 98183 DEBUG oslo_service.
2016-04-07 11:28:19.744 98183 DEBUG oslo_service.
2016-04-07 11:28:19.744 98183 DEBUG oslo_service.
2016-04-07 11:28:19.753 98183 ERROR oslo.messaging.
2016-04-07 11:28:20.759 98183 ERROR oslo.messaging.
2016-04-07 11:28:22.767 98183 ERROR oslo.messaging.
2016-04-07 11:28:26.778 98183 ERROR oslo.messaging.
2016-04-07 11:28:32.789 98183 ERROR oslo.messaging.
2016-04-07 11:28:40.802 98183 ERROR oslo.messaging.
2016-04-07 11:28:50.818 98183 ERROR oslo.messaging.
2016-04-07 11:29:02.835 98183 ERROR oslo.messaging.
2016-04-07 11:29:16.857 98183 ERROR oslo.messaging.
so this the problem in your log seems not because aodh-notifier dosen't retry to connect rabbitmq.