Failed listener initialization causes unhandled exception during stop

Bug #1785048 reported by Andras Kovi on 2018-08-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
Undecided
Andras Kovi

Bug Description

Unfortunately, it is not clear if this is causing further issues but as least it pollutes the logs with irrelevant errors. However, if the unhandled error can cause the sys.exit() not happen, then this is an even more serious issue.

Here is an example log excerpt:

2018-07-31T03:38:32.816398+02:00 localhost service: service Error starting thread.
Traceback (most recent call last):
 File "/opt/venv/lib/python2.7/site-packages/oslo_service/service.py", line 792, in run_service
 service.start()
 File "/opt/venv/lib/python2.7/site-packages/mistral/executors/executor_server.py", line 59, in start
 self._rpc_server.run(executor='threading')
 File "/opt/venv/lib/python2.7/site-packages/mistral/rpc/oslo/oslo_server.py", line 56, in run
 self.oslo_server.start()
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper
 log_after, timeout_timer)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once
 post_fn = fn()
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda>
 states[state].run_once(lambda: fn(self, *args, **kwargs),
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 416, in start
 self.listener = self._create_listener()
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 151, in _create_listener
 return self.transport._listen(self._target, 1, None)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/transport.py", line 148, in _listen
 batch_timeout)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 597, in listen
 callback=listener)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1120, in declare_topic_consumer
 self.declare_consumer(consumer)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1023, in declare_consumer
 error_callback=_connect_error)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure
 raise exceptions.MessageDeliveryFailure(msg)
MessageDeliveryFailure: Unable to connect to AMQP server on rmq:5671 after None tries: Queue.bind: (404) NOT_FOUND - no binding mistral_executor between exchange 'openstack' in vhost '/' and queue 'mistral_executor' in vhost '/'
2018-07-31T03:38:32.817689+02:00 localhost service: service Unhandled error
Traceback (most recent call last):
 File "/opt/venv/bin/mistral-server", line 11, in <module>
 sys.exit(main())
 File "/opt/venv/lib/python2.7/site-packages/mistral/cmd/launch.py", line 213, in main
 launch_any(set(cfg.CONF.server))
 File "/opt/venv/lib/python2.7/site-packages/mistral/cmd/launch.py", line 119, in launch_any
 SERVER_THREAD_MANAGER.wait()
 File "/opt/venv/lib/python2.7/site-packages/oslo_service/service.py", line 392, in wait
 status, signo = self._wait_for_exit_or_signal()
 File "/opt/venv/lib/python2.7/site-packages/oslo_service/service.py", line 377, in _wait_for_exit_or_signal
 self.stop()
 File "/opt/venv/lib/python2.7/site-packages/oslo_service/service.py", line 292, in stop
 self.services.stop()
 File "/opt/venv/lib/python2.7/site-packages/oslo_service/service.py", line 758, in stop
 service.stop()
 File "/opt/venv/lib/python2.7/site-packages/mistral/executors/executor_server.py", line 70, in stop
 self._rpc_server.stop(graceful)
 File "/opt/venv/lib/python2.7/site-packages/mistral/rpc/oslo/oslo_server.py", line 59, in stop
 self.oslo_server.stop()
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper
 log_after, timeout_timer)
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once
 post_fn = fn()
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda>
 states[state].run_once(lambda: fn(self, *args, **kwargs),
 File "/opt/venv/lib/python2.7/site-packages/oslo_messaging/server.py", line 431, in stop
 self.listener.stop()
AttributeError: 'NoneType' object has no attribute 'stop'
2018-07-31T03:38:33.117401+02:00 localhost service: service Failed to declare consumer for topic 'mistral_executor': Queue.bind: (404) NOT_FOUND - no binding mistral_executor between exchange 'openstack' in vhost '/' and queue 'mistral_executor' in vhost '/'

Fix proposed to branch: master
Review: https://review.openstack.org/588217

Changed in oslo.messaging:
assignee: nobody → Andras Kovi (akovi)
status: New → In Progress

Reviewed: https://review.openstack.org/588217
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=d8fa6d5edfa171ae5f36b8d1269f87716830ba7b
Submitter: Zuul
Branch: master

commit d8fa6d5edfa171ae5f36b8d1269f87716830ba7b
Author: Andras Kovi <email address hidden>
Date: Thu Aug 2 13:24:11 2018 +0200

    Call listener stop only if listener is initialized

    Change-Id: I1303d74af7a242edefe774d8f5e12e63614c0158
    Closes-Bug: #1785048

Changed in oslo.messaging:
status: In Progress → Fix Released

This issue was fixed in the openstack/oslo.messaging 9.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers