| 2019-12-27 08:33:03 |
Wave Wang |
description |
Hi,
When running for some days, launch audit periodly and execute continuous audit stopped. Then when I create new continuous its state stuck at pending.Here is the log when background thread stopped.
watcher-decision-engine.log
2019-12-24 12:16:55.072 ERROR oslo_db.sqlalchemy.exc_filters [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] DB exception wrapped.: ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters return self._revalidate_connection()
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters raise exc.ResourceClosedError("This Connection is closed")
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters
2019-12-24 12:16:56.719 ERROR oslo_service.threadgroup [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] Error waiting on thread.: IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup Traceback (most recent call last):
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 116, in _perform_action_on_threads
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup action_func(x)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 160, in <lambda>
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup lambda x: x.wait(),
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 57, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.thread.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self._exit_event.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return hubs.get_hub().switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.greenlet.switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in run
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.fire_timers(self.clock())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.squelch_timer_exception(timer, sys.exc_info())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 385, in squelch_timer_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup traceback.print_exception(*exc_info)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 124, in print_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup _print(file, 'Traceback (most recent call last):')
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 13, in _print
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup file.write(str+terminator)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup
How to produce: Dont know |
Hi,
When running for some days, launch audit periodly and execute continuous audit stopped. Then when I create new continuous its state stuck at pending.I think maybe log below will do some help.
watcher-decision-engine.log
2019-12-24 12:16:55.072 ERROR oslo_db.sqlalchemy.exc_filters [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] DB exception wrapped.: ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters return self._revalidate_connection()
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters raise exc.ResourceClosedError("This Connection is closed")
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters
2019-12-24 12:16:56.719 ERROR oslo_service.threadgroup [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] Error waiting on thread.: IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup Traceback (most recent call last):
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 116, in _perform_action_on_threads
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup action_func(x)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 160, in <lambda>
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup lambda x: x.wait(),
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 57, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.thread.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self._exit_event.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return hubs.get_hub().switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.greenlet.switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in run
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.fire_timers(self.clock())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.squelch_timer_exception(timer, sys.exc_info())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 385, in squelch_timer_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup traceback.print_exception(*exc_info)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 124, in print_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup _print(file, 'Traceback (most recent call last):')
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 13, in _print
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup file.write(str+terminator)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup
How to produce: Dont know |
|
