task stuck in RUNNING state when all action executions are finished

[Nikolay Makhotkin]

once in a while the task which has 'with-items' stuck in RUNNING state when, in fact, all its action-executions are completed.

It can be something wrong with DB transactions, when, say, task is created 4 action executions and when 4th action execution is being completed, it can see only 2 of 4 as succeed but it should see all of them. (The result of debugging).

[Renat Akhmerov (rakhmerov)]

Looks like HA and using "with-items" increase chances of reproducing this bug.

[Renat Akhmerov (rakhmerov)]

Related to https://bugs.launchpad.net/mistral/+bug/1518012

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/251370

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253819
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=976667c0a22f604559e19135f30dea803e3080f6
Submitter: Jenkins
Branch: master

commit 976667c0a22f604559e19135f30dea803e3080f6
Author: Moshe Elisha <email address hidden>
Date: Sat Dec 5 15:31:24 2015 +0000

    Fix concurrency issues by using READ_COMMITTED

    The state of a workflow execution was not updated even when all task
    executions were completed if some tasks finished at the same time as
    other tasks.
    Because we were using our connections with transaction isolation
    level = REPEATABLE_READ - Each process was using a snapshot of the DB
    created at the first read statement in that transaction.
    When a task finished and evaluated the state of all the other tasks
    it did not see the up-to-date state of those tasks - and so, because
    not all tasks were completed - the task did not change the workflow
    execution state.

    Similar behavior happened with multiple action executions under same
    task. On completion, each action execution checked the status of the
    other action executions and did not see the up-to-date state of these
    action execution - causing task execution to stay in RUNNING state.

    Change-Id: I12f66134d92b8ed39df9d6128d7de5ee49aa8623
    Closes-Bug: #1518012
    Closes-Bug: #1513456

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/255718

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/255718
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=09f206f863396591c8a0b0d1805a72ae8b3966cd
Submitter: Jenkins
Branch: stable/liberty

commit 09f206f863396591c8a0b0d1805a72ae8b3966cd
Author: Moshe Elisha <email address hidden>
Date: Sat Dec 5 15:31:24 2015 +0000

    Fix concurrency issues by using READ_COMMITTED

    The state of a workflow execution was not updated even when all task
    executions were completed if some tasks finished at the same time as
    other tasks.
    Because we were using our connections with transaction isolation
    level = REPEATABLE_READ - Each process was using a snapshot of the DB
    created at the first read statement in that transaction.
    When a task finished and evaluated the state of all the other tasks
    it did not see the up-to-date state of those tasks - and so, because
    not all tasks were completed - the task did not change the workflow
    execution state.

    Similar behavior happened with multiple action executions under same
    task. On completion, each action execution checked the status of the
    other action executions and did not see the up-to-date state of these
    action execution - causing task execution to stay in RUNNING state.

    Change-Id: I12f66134d92b8ed39df9d6128d7de5ee49aa8623
    Closes-Bug: #1518012
    Closes-Bug: #1513456
    (cherry picked from commit 976667c0a22f604559e19135f30dea803e3080f6)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dawid Deja (<email address hidden>) on branch: master
Review: https://review.openstack.org/251370

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/mistral 2.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/mistral 1.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/mistral 1.0.1 release.