masakari

notification stuck into running status when timeout

Bug #1996835 reported by suzhengwei on 2022-11-17

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	masakari	Fix Released	High	suzhengwei

Bug Description

In the env, there is an VM failure notification stuck into running status. It's pretty obvious that IO_ERROR happens to the instance and it executed StopInstanceTask workflow timeout, which results in this problem.

Log as following,

2022-11-16 15:44:35.499 6 INFO masakari.engine.manager [None req-35728731-78a9-4a11-aa74-8359061cb85d 1104d0ae28734c0a8a70c941ad4a6400 befe712fa18349e1abed7b77bf3fbea2 - - -] Processing notification 961d8738-ad57-4dd5-96f8-745e3cda89c9 of type: VM
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder [req-35728731-78a9-4a11-aa74-8359061cb85d req-5df7d477-f163-479b-99cd-8dbb0e482183 nova - - - -] Engine 'masakari.engine.drivers.taskflow.instance_failure.StopInstanceTask;instance:recovery==1.0' atom post-completion failed: taskflow.exceptions.StorageFailure: Storage backend internal error
  TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder Traceback (most recent call last):
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 338, in _memory_lock
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder yield
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder yield
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder ignore_missing=ignore_missing)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 152, in _update_object
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self._set_item(path, self._serialize(obj), transaction)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 68, in _serialize
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder return base._format_atom(obj)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/base.py", line 127, in _format_atom
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder 'atom': atom_detail.to_dict(),
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/models.py", line 627, in to_dict
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder failure = self.failure.to_dict()
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 517, in to_dict
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder 'causes': [f.to_dict() for f in self.causes],
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 424, in causes
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self._causes = tuple(self._extract_causes_iter(self.exception))
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 406, in _extract_causes_iter
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder yield cls(exc_info=exc_info)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 198, in __init__
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder % (exc_info[0], type(exc_info[0])))
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder The above exception was the direct cause of the following exception:
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder Traceback (most recent call last):
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/builder.py", line 231, in complete_an_atom
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder do_complete(atom, outcome, result)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/completer.py", line 168, in complete
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder handler.complete_execution(node, result)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 117, in complete_execution
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self.change_state(task, states.FAILURE, result=result)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 65, in change_state
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self._storage.save(task.name, save_result, state)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/fasteners/lock.py", line 85, in wrapper
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder return f(self, *args, **kwargs)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 633, in save
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self._with_connection(self._save_atom_detail, source, clone)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 344, in _with_connection
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder return functor(conn, *args, **kwargs)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 491, in _save_atom_detail
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder original_atom_detail.update(conn.update_atom_details(atom_detail))
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder ignore_missing=ignore_missing)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder yield
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 343, in _memory_lock
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder "Storage backend internal error")
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/exceptions.py", line 52, in raise_with_cause
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder File "<string>", line 3, in raise_from
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder taskflow.exceptions.StorageFailure: Storage backend internal error
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.476 6 ERROR taskflow.engines.action_engine.builder
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine [req-35728731-78a9-4a11-aa74-8359061cb85d req-5df7d477-f163-479b-99cd-8dbb0e482183 nova - - - -] Engine execution has failed, something bad must of happened (last 10 machine transitions were [('UNDEFINED', 'RESUMING'), ('RESUMING', 'SCHEDULING'), ('SCHEDULING', 'WAITING'), ('WAITING', 'ANALYZING'), ('ANALYZING', 'GAME_OVER'), ('GAME_OVER', 'FAILURE')]): taskflow.exceptions.StorageFailure: Storage backend internal error
  TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine Traceback (most recent call last):
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 338, in _memory_lock
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine yield
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine yield
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine ignore_missing=ignore_missing)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 152, in _update_object
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self._set_item(path, self._serialize(obj), transaction)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 68, in _serialize
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine return base._format_atom(obj)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/base.py", line 127, in _format_atom
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine 'atom': atom_detail.to_dict(),
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/models.py", line 627, in to_dict
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine failure = self.failure.to_dict()
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 517, in to_dict
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine 'causes': [f.to_dict() for f in self.causes],
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 424, in causes
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self._causes = tuple(self._extract_causes_iter(self.exception))
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 406, in _extract_causes_iter
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine yield cls(exc_info=exc_info)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 198, in __init__
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine % (exc_info[0], type(exc_info[0])))
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine The above exception was the direct cause of the following exception:
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine Traceback (most recent call last):
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 300, in run_iter
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine failure.Failure.reraise_if_any(memory.failures)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine failures[0].reraise()
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine six.reraise(*self._exc_info)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine raise value
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/builder.py", line 231, in complete_an_atom
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine do_complete(atom, outcome, result)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/completer.py", line 168, in complete
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine handler.complete_execution(node, result)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 117, in complete_execution
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self.change_state(task, states.FAILURE, result=result)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 65, in change_state
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self._storage.save(task.name, save_result, state)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/fasteners/lock.py", line 85, in wrapper
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine return f(self, *args, **kwargs)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 633, in save
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self._with_connection(self._save_atom_detail, source, clone)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 344, in _with_connection
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine return functor(conn, *args, **kwargs)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 491, in _save_atom_detail
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine original_atom_detail.update(conn.update_atom_details(atom_detail))
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine ignore_missing=ignore_missing)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine yield
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 343, in _memory_lock
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine "Storage backend internal error")
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/exceptions.py", line 52, in raise_with_cause
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine File "<string>", line 3, in raise_from
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine taskflow.exceptions.StorageFailure: Storage backend internal error
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.499 6 ERROR taskflow.engines.action_engine.engine
2022-11-16 15:45:41.516 6 WARNING masakari.engine.drivers.taskflow.driver [req-35728731-78a9-4a11-aa74-8359061cb85d req-5df7d477-f163-479b-99cd-8dbb0e482183 nova - - - -] Flow 'instance_recovery_engine' (8908d32e-cad6-45cf-9e09-06d7307600cb) transitioned into state 'FAILURE' from state 'RUNNING': taskflow.exceptions.StorageFailure: Storage backend internal error
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server [req-35728731-78a9-4a11-aa74-8359061cb85d req-5df7d477-f163-479b-99cd-8dbb0e482183 nova - - - -] Exception during message handling: taskflow.exceptions.StorageFailure: Storage backend internal error
  TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 338, in _memory_lock
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server yield
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server yield
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server ignore_missing=ignore_missing)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 152, in _update_object
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._set_item(path, self._serialize(obj), transaction)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 68, in _serialize
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server return base._format_atom(obj)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/base.py", line 127, in _format_atom
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server 'atom': atom_detail.to_dict(),
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/models.py", line 627, in to_dict
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server failure = self.failure.to_dict()
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 517, in to_dict
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server 'causes': [f.to_dict() for f in self.causes],
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 424, in causes
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._causes = tuple(self._extract_causes_iter(self.exception))
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 406, in _extract_causes_iter
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server yield cls(exc_info=exc_info)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 198, in __init__
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server % (exc_info[0], type(exc_info[0])))
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server The above exception was the direct cause of the following exception:
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/engine/manager.py", line 396, in process_notification
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._process_notification(context, notification)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/engine/manager.py", line 384, in _process_notification
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server do_process_notification(notification)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/utils.py", line 270, in inner
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/engine/manager.py", line 372, in do_process_notification
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server context, notification)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/engine/manager.py", line 161, in _handle_notification_type_instance
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server notification.notification_uuid)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/masakari/engine/drivers/taskflow/driver.py", line 187, in execute_instance_failure
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server flow_engine.run()
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 328, in run_iter
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._change_state(states.FAILURE)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server raise value
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 300, in run_iter
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(memory.failures)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server failures[0].reraise()
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server raise value
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/builder.py", line 231, in complete_an_atom
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server do_complete(atom, outcome, result)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/completer.py", line 168, in complete
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server handler.complete_execution(node, result)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 117, in complete_execution
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self.change_state(task, states.FAILURE, result=result)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/engines/action_engine/actions/task.py", line 65, in change_state
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._storage.save(task.name, save_result, state)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/fasteners/lock.py", line 85, in wrapper
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 633, in save
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self._with_connection(self._save_atom_detail, source, clone)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 344, in _with_connection
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server return functor(conn, *args, **kwargs)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/storage.py", line 491, in _save_atom_detail
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server original_atom_detail.update(conn.update_atom_details(atom_detail))
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/path_based.py", line 224, in update_atom_details
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server ignore_missing=ignore_missing)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 373, in _transaction
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server yield
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server self.gen.throw(type, value, traceback)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/persistence/backends/impl_memory.py", line 343, in _memory_lock
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server "Storage backend internal error")
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/taskflow/exceptions.py", line 52, in raise_with_cause
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server File "<string>", line 3, in raise_from
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server taskflow.exceptions.StorageFailure: Storage backend internal error
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server TypeError: Invalid exception type '<class 'eventlet.timeout.Timeout'>' (<class 'type'>)
2022-11-16 15:45:41.517 6 ERROR oslo_messaging.rpc.server

Tags:

suzhengwei (sue.sam) on 2022-11-17

Changed in masakari:
assignee:	nobody → suzhengwei (sue.sam)
importance:	Undecided → High
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-17: Fix proposed to masakari (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/masakari/+/864807

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-06: Fix merged to masakari (master)

Reviewed: https://review.opendev.org/c/openstack/masakari/+/864807
Committed: https://opendev.org/openstack/masakari/commit/7ec3edda1ada9c2464d79c84b0fd1d1be22f9336
Submitter: "Zuul (22348)"
Branch: master

commit 7ec3edda1ada9c2464d79c84b0fd1d1be22f9336
Author: sue <sugar-2008@163.com>
Date: Thu Nov 17 10:26:15 2022 +0800

Fix notfication stuck into running status when timeout

For instance or process failure workflow, the failure notification
would stuck into running status if timeout.

Closes-Bug: #1996835
Change-Id: I61e941ab9dd831369fcc46a132ae2b11c1dd23ba

Changed in masakari:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix proposed to masakari (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/masakari/+/866662

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix proposed to masakari (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/masakari/+/866663

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix proposed to masakari (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/masakari/+/866664

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix merged to masakari (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/masakari/+/866664
Committed: https://opendev.org/openstack/masakari/commit/4f6a740c2c40729122446a2766127363799df2b0
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4f6a740c2c40729122446a2766127363799df2b0
Author: sue <sugar-2008@163.com>
Date: Thu Nov 17 10:26:15 2022 +0800

Fix notfication stuck into running status when timeout

For instance or process failure workflow, the failure notification
would stuck into running status if timeout.

    Closes-Bug: #1996835
    Change-Id: I61e941ab9dd831369fcc46a132ae2b11c1dd23ba
    (cherry picked from commit 7ec3edda1ada9c2464d79c84b0fd1d1be22f9336)

tags:	added: in-stable-xena
tags:	added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix merged to masakari (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/masakari/+/866663
Committed: https://opendev.org/openstack/masakari/commit/9d1b9e3e9bd2d88749eae279d65149301f3db5a9
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 9d1b9e3e9bd2d88749eae279d65149301f3db5a9
Author: sue <sugar-2008@163.com>
Date: Thu Nov 17 10:26:15 2022 +0800

Fix notfication stuck into running status when timeout

For instance or process failure workflow, the failure notification
would stuck into running status if timeout.

    Closes-Bug: #1996835
    Change-Id: I61e941ab9dd831369fcc46a132ae2b11c1dd23ba
    (cherry picked from commit 7ec3edda1ada9c2464d79c84b0fd1d1be22f9336)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-12-07: Fix merged to masakari (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/masakari/+/866662
Committed: https://opendev.org/openstack/masakari/commit/e496aad0ecc84beb3b000019bf45b244e6bf5fa6
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e496aad0ecc84beb3b000019bf45b244e6bf5fa6
Author: sue <sugar-2008@163.com>
Date: Thu Nov 17 10:26:15 2022 +0800

Fix notfication stuck into running status when timeout

For instance or process failure workflow, the failure notification
would stuck into running status if timeout.

    Closes-Bug: #1996835
    Change-Id: I61e941ab9dd831369fcc46a132ae2b11c1dd23ba
    (cherry picked from commit 7ec3edda1ada9c2464d79c84b0fd1d1be22f9336)