Hi,
When running for some days, launch audit periodly and execute continuous audit stopped. Then when I create new continuous its state stuck at pending.Here is the log when background thread stopped.
watcher-decision-engine.log
2019-12-24 12:16:55.072 ERROR oslo_db.sqlalchemy.exc_filters [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] DB exception wrapped.: ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 352, in connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters return self._revalidate_connection()
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 432, in _revalidate_connection
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters raise exc.ResourceClosedError("This Connection is closed")
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters ResourceClosedError: This Connection is closed
2019-12-24 12:16:55.072 TRACE oslo_db.sqlalchemy.exc_filters
2019-12-24 12:16:56.719 ERROR oslo_service.threadgroup [req-0547a92a-4347-47e6-b0ab-a34682e99048 None None] Error waiting on thread.: IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup Traceback (most recent call last):
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 116, in _perform_action_on_threads
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup action_func(x)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 160, in <lambda>
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup lambda x: x.wait(),
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/oslo_service/threadgroup.py", line 57, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.thread.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self._exit_event.wait()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return hubs.get_hub().switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup return self.greenlet.switch()
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 336, in run
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.fire_timers(self.clock())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 461, in fire_timers
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup self.squelch_timer_exception(timer, sys.exc_info())
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/openstack/venvs/watcher-4.4.0.20191213/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 385, in squelch_timer_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup traceback.print_exception(*exc_info)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 124, in print_exception
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup _print(file, 'Traceback (most recent call last):')
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup File "/usr/lib64/python2.7/traceback.py", line 13, in _print
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup file.write(str+terminator)
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup IOError: [Errno 32] Broken pipe
2019-12-24 12:16:56.719 TRACE oslo_service.threadgroup
Hi,
When running for some days, launch audit periodly and execute continuous audit stopped. Then when I create new continuous its state stuck at pending.Here is the log when background thread stopped.
watcher- decision- engine. log
2019-12-24 12:16:55.072 ERROR oslo_db. sqlalchemy. exc_filters [req-0547a92a- 4347-47e6- b0ab-a34682e990 48 None None] DB exception wrapped.: ResourceClosedE rror: This Connection is closed sqlalchemy. exc_filters Traceback (most recent call last): sqlalchemy. exc_filters File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ sqlalchemy/ engine/ base.py" , line 352, in connection sqlalchemy. exc_filters return self._revalidat e_connection( ) sqlalchemy. exc_filters File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ sqlalchemy/ engine/ base.py" , line 432, in _revalidate_ connection sqlalchemy. exc_filters raise exc.ResourceClo sedError( "This Connection is closed") sqlalchemy. exc_filters ResourceClosedE rror: This Connection is closed sqlalchemy. exc_filters threadgroup [req-0547a92a- 4347-47e6- b0ab-a34682e990 48 None None] Error waiting on thread.: IOError: [Errno 32] Broken pipe threadgroup Traceback (most recent call last): threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ oslo_service/ threadgroup. py", line 116, in _perform_ action_ on_threads threadgroup action_func(x) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ oslo_service/ threadgroup. py", line 160, in <lambda> threadgroup lambda x: x.wait(), threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ oslo_service/ threadgroup. py", line 57, in wait threadgroup return self.thread.wait() threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ greenthread. py", line 175, in wait threadgroup return self._exit_ event.wait( ) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ event.py" , line 121, in wait threadgroup return hubs.get_ hub().switch( ) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ hubs/hub. py", line 294, in switch threadgroup return self.greenlet. switch( ) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ hubs/hub. py", line 336, in run threadgroup self.fire_ timers( self.clock( )) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ hubs/hub. py", line 461, in fire_timers threadgroup self.squelch_ timer_exception (timer, sys.exc_info()) threadgroup File "/openstack/ venvs/watcher- 4.4.0.20191213/ lib/python2. 7/site- packages/ eventlet/ hubs/hub. py", line 385, in squelch_ timer_exception threadgroup traceback. print_exception (*exc_info) threadgroup File "/usr/lib64/ python2. 7/traceback. py", line 124, in print_exception threadgroup _print(file, 'Traceback (most recent call last):') threadgroup File "/usr/lib64/ python2. 7/traceback. py", line 13, in _print threadgroup file.write( str+terminator) threadgroup IOError: [Errno 32] Broken pipe threadgroup
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:55.072 TRACE oslo_db.
2019-12-24 12:16:56.719 ERROR oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
2019-12-24 12:16:56.719 TRACE oslo_service.
How to produce: Dont know