Endless loop on StorageFailure

[Ann Taraday]

If an error occures with writing atom detail in database(
persistence backend) flow execution enters an endless loop
throwing errors and retrying to save details.

020-07-31 12:33:46.780 28095 ERROR taskflow.storage Traceback (most recent call last):
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/taskflow/persistence/backends/impl_sqlalchemy.py", line 432, in upd
ate_atom_details
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage self._update_atom_details(conn, atom_detail, e_ad)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/taskflow/persistence/backends/impl_sqlalchemy.py", line 456, in _up
date_atom_details
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage .values(e_ad.to_dict()))
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1014, in execute
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage return meth(self, multiparams, params)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage return connection._execute_clauseelement(self, multiparams, params)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage distilled_params,
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1208, in _execute_context
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage e, util.text_type(statement), parameters, None, None
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1512, in _handle_dbapi_exception
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage sqlalchemy_exception, with_traceback=exc_info[2], from_=e
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 178, in raise_
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage raise exception
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1205, in _execute_context
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage context = constructor(dialect, self, conn, *args)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 865, in _init_compiled
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage for key in compiled_params
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 865, in <genexpr>
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage for key in compiled_params
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/type_api.py", line 1230, in process
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage return impl_processor(process_param(value, dialect))
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/local/lib/python3.6/dist-packages/sqlalchemy_utils/types/json.py", line 80, in process_bind_param
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage value = six.text_type(json.dumps(value))
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage return _default_encoder.encode(obj)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage chunks = self.iterencode(o, _one_shot=True)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage return _iterencode(o, 0)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage File "/usr/lib/python3.6/json/encoder.py", line 180, in default
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage o.__class__.__name__)
2020-07-31 12:33:46.780 28095 ERROR taskflow.storage sqlalchemy.exc.StatementError: (builtins.TypeError) Object of type 'Amphora' is not JSON serializable

Also if task was set to create some resources - it may spawn them endlessly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744156
Committed: https://git.openstack.org/cgit/openstack/taskflow/commit/?id=c32454213e5aa1b6cda770f56ddef18f06c2c3c2
Submitter: Zuul
Branch: master

commit c32454213e5aa1b6cda770f56ddef18f06c2c3c2
Author: Ann Taraday <email address hidden>
Date: Fri Jul 31 16:58:23 2020 +0400

    Avoid endless loop on StorageFailure

    If an error occures with writing atom detail in database(
    persistence backend) flow execution enters an endless loop
    throwing errors and retrying to save details.

    Avoid this situation and log exception message.

    Change-Id: Ic6b0a78d20124cc027468ecc6aeff189c25d1a8a
    Closes-bug: 1889773