Workflow execution gets stuck in 'RUNNING' state if DB error occurs during updating DB after outputs is evaluated. Mistral keeps retrying for 50 times (as per db retry policy) and still fails.
Setup details:
MySQL v5.6 with InnoDB engine
Mistral master
[engine]
execution_field_size_limit_kb = -1
Bug description:
The DB errors faced in case of saving large outputs (say 16MB in size) can be due to one or more of following reasons:
1) `max_packet_allowed` is less than what is required to handle such large size row update,
2) Large UPDATE queries fail due to insufficient redo size log (size of update data is > 10% of redo size log) see bug: https://bugs.mysql.com/bug.php?id=69477
Bug from mistral side is improper error handling --- Mistral keeps retrying operation (assuming error to be recoverable Deadlock or similar).
How to reproduce:
Execute a WF that has very large size outputs.
Workflow:
---
version: "2.0"
bugs_me:
output:
big_dict: <% 2097152 * ["ABCD"] %>
tasks:
t:
action: std.noop
Expectation:
Workflow status should be 'ERROR'.
the exception raised in this case is very generic, as it is raised in other cases too in which we want it to retry.
for example, in case of a timeout.
the exception raised in case of a Mysql is 'MySQL server has gone away' , which can happen for many reasons, incorrect package, timeout or because of a package that is too big.