UI autotests failed because of DB deadlocks in the fake tasks

Bug #1653083 reported by Julia Aranovich
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Georgy Kibardin
Mitaka
Fix Released
High
Georgy Kibardin
Newton
Fix Committed
High
Georgy Kibardin
summary: - ui autotests failed because of fake tasks
+ UI autotests failed because of DB deadlocks in the fake tasks
Changed in fuel:
importance: Undecided → High
tags: added: non-release
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Georgy Kibardin (gkibardin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/426675

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/426675
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=ff04eb8394d81750608a95bedf6ab8cd5244d5e8
Submitter: Jenkins
Branch: master

commit ff04eb8394d81750608a95bedf6ab8cd5244d5e8
Author: Georgy Kibardin <email address hidden>
Date: Mon Jan 30 11:47:50 2017 +0300

    Moving task cache update into task creation transaction

    Doing this we avoid inclusion of task cache update statement into the next
    transaction which may cause different problems such as deadlock.
    In this particular case we've got the following deadlock:
    1. DeleteIBPImagesTask makes UPDATE tasks SET cache....
    2. Response handler in receiver makes SELECT clusters FOR UPDATE
    3. The code following DeleteIBPImagesTask makes SELECT clusters FOR UPDATE
    4. Response handler performs SELECTS tasks FOR UPDATE

    Change-Id: Ic8e5f2386364421b0667f920499e031f587f726e
    Closes-Bug: #1653083

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/430381

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/430382

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/newton)

Reviewed: https://review.openstack.org/430381
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=ea0f198757d1ee5d5190c905cc9dd6c632cf889a
Submitter: Jenkins
Branch: stable/newton

commit ea0f198757d1ee5d5190c905cc9dd6c632cf889a
Author: Georgy Kibardin <email address hidden>
Date: Mon Jan 30 11:47:50 2017 +0300

    Moving task cache update into task creation transaction

    Doing this we avoid inclusion of task cache update statement into the next
    transaction which may cause different problems such as deadlock.
    In this particular case we've got the following deadlock:
    1. DeleteIBPImagesTask makes UPDATE tasks SET cache....
    2. Response handler in receiver makes SELECT clusters FOR UPDATE
    3. The code following DeleteIBPImagesTask makes SELECT clusters FOR UPDATE
    4. Response handler performs SELECTS tasks FOR UPDATE

    Change-Id: Ic8e5f2386364421b0667f920499e031f587f726e
    Closes-Bug: #1653083
    (cherry picked from commit ff04eb8394d81750608a95bedf6ab8cd5244d5e8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/mitaka)

Reviewed: https://review.openstack.org/430382
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=297c6c9f790c08d88e5af39de1fa83ce3f27e07b
Submitter: Jenkins
Branch: stable/mitaka

commit 297c6c9f790c08d88e5af39de1fa83ce3f27e07b
Author: Georgy Kibardin <email address hidden>
Date: Mon Jan 30 11:47:50 2017 +0300

    Moving task cache update into task creation transaction

    Doing this we avoid inclusion of task cache update statement into the next
    transaction which may cause different problems such as deadlock.
    In this particular case we've got the following deadlock:
    1. DeleteIBPImagesTask makes UPDATE tasks SET cache....
    2. Response handler in receiver makes SELECT clusters FOR UPDATE
    3. The code following DeleteIBPImagesTask makes SELECT clusters FOR UPDATE
    4. Response handler performs SELECTS tasks FOR UPDATE

    Change-Id: Ic8e5f2386364421b0667f920499e031f587f726e
    Closes-Bug: #1653083
    (cherry picked from commit ff04eb8394d81750608a95bedf6ab8cd5244d5e8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-web 11.0.0.0rc1

This issue was fixed in the openstack/fuel-web 11.0.0.0rc1 release candidate.

Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.2 + mu1 updates.

I've checked screenshots of the latest runs of job https://ci.fuel-infra.org/job/verify-fuel-ui-on-fuel-web (from Mar 7, 2017) and there is no any run with deadlock issue.

tags: added: on-verification
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.