After-task logic runs multiple times if tasks run in parallel
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Mistral |
Fix Released
|
High
|
Renat Akhmerov |
Bug Description
When executing a workflow with multiple tasks (let's mark them as task_a, task_b and task_c ). where some tasks run async actions, such as std.mistral_http (task_a), and contain "on-complete" sections . The tasks the should run later (in the "on-complete" section) (task_c) sometimes run more times then intended.
These tasks are ran from the actual completed task (task_a), but if a different task (task_b) is completed at the same time, it sometimes runs the task (task_c) again.
This seems to happen since both tasks (task_a and task_b) reach the method _on_task_
I reproduced this issue using this workflow:
---
version: '2.0'
test_tasks:
tasks:
task1:
action: std.mistral_http url="http://
on-complete:
- task4
task2:
action: std.mistral_http url="http://
on-complete:
- task5
task3:
action: std.mistral_http url="http://
on-complete:
- task6
task4:
action: std.echo output='TASK4'
task5:
action: std.echo output='TASK5'
task6:
action: std.echo output='TASK6'
and running execution-update --state SUCCESS for task1, task2, and task3's action-executions concurrently.
this resulted in:
task 4 running twice: once when task1 finished, and once when task 5 finished.
task 5 running twice: once when task 2 finished and once when task 3 finished.
task 6 running twice: once when task 3 finished and once when task 1 finished.
summary: |
- on-complete tasks are ran multiple times + on-complete tasks are ran multiple times on workflow resume |
summary: |
- on-complete tasks are ran multiple times on workflow resume + After-task logic runs multiple times if tasks run in parallel |
Changed in mistral: | |
importance: | Undecided → High |
Changed in mistral: | |
assignee: | nobody → Renat Akhmerov (rakhmerov) |
milestone: | none → mitaka-2 |
Changed in mistral: | |
milestone: | mitaka-2 → mitaka-3 |
Changed in mistral: | |
status: | New → In Progress |
Changed in mistral: | |
milestone: | mitaka-3 → mitaka-rc1 |
Changed in mistral: | |
milestone: | mitaka-rc1 → mitaka-rc2 |
Changed in mistral: | |
milestone: | mitaka-rc2 → 2.0.0 |
I'm attaching the logs from the bug reproduction.
I have two mistral-engine logs, since we run mistral in HA.
this is the output of the final task list of the execution:
# mistral task-list 54bfc592- 40e9-4466- b115-87344b91ac 91 ------- ------- ------- ------- ----+-- -----+- ------- ------- +------ ------- ------- ------- ------- ----+-- ------- +------ ------+ ------- ------- ------- ------- ----+-- -----+- ------- ------- +------ ------- ------- ------- ------- ----+-- ------- +------ ------+ bd20-46f6- a1ca-06d4a5e3dd f7 | task1 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | dba7-4d7e- 8811-f9659c6821 f2 | task2 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | a824-4446- 8063-24e7d40c17 3a | task3 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | 9ac7-46d3- a2af-6d986773b5 af | task5 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | b5de-4ae4- b087-1dee7e93b9 52 | task5 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | 9c92-4238- 840e-082f877fb5 65 | task6 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | 22f8-42d1- b504-8fe15e39de 34 | task4 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | 79af-4b09- 8c64-bc00ddfe36 e6 | task6 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | afc1-44ff- b6de-bdb9ce5edc f4 | task4 | test_tasks | 54bfc592- 40e9-4466- b115-87344b91ac 91 | SUCCESS | None | ------- ------- ------- ------- ----+-- -----+- ------- ------- +------ ------- ------- ------- ------- ----+-- ------- +------ ------+
+------
| ID | Name | Workflow name | Execution ID | State | State info |
+------
| 8dca3757-
| 8286e9db-
| 9dc2cf86-
| 30ca8a3f-
| a72c14c9-
| c78e2f2f-
| b2b3cd2c-
| 9530ebc6-
| 67d49e3f-
+------