task_state may be overwritten in race condition

Bug #1037372 reported by Yun Mao
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Yun Mao

Bug Description

task_state acts like a guard to avoid concurrent tasks to be scheduled. There might be two race conditions:

1) two tasks are concurrently accepted by api, check task_state to be None and allow the tasks to be executed concurrently.

2) one ordinary task is running, so that task_state is set. The delete task is accepted at API, and will "take over" and change task_state to DELETING. However the first task may continue to update task_state or set it to None as it finishes.

Changed in nova:
assignee: nobody → Yun Mao (yunmao)
importance: Undecided → High
status: New → Triaged
milestone: none → folsom-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/11895

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/11895
Committed: http://github.com/openstack/nova/commit/85b9ee6b5b74d15a6455cb6632578bea07d8eb11
Submitter: Jenkins
Branch: master

commit 85b9ee6b5b74d15a6455cb6632578bea07d8eb11
Author: Yun Mao <email address hidden>
Date: Thu Aug 23 11:13:58 2012 -0400

    Make instance_update_and_get_original() atomic

    Use the same session object within a transaction to retrieve
    and update the instance object, including metadata and system
    metadata.

    In preparation to fix bug 1037372

    Change-Id: I886da12bb85f60c2b23609a35b6d9a43040216ce

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12222

Changed in nova:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/12222
Committed: http://github.com/openstack/nova/commit/4082c8375a6ae7e7e67c7ad2e263be2d5fc3dd1f
Submitter: Jenkins
Branch: master

commit 4082c8375a6ae7e7e67c7ad2e263be2d5fc3dd1f
Author: Yun Mao <email address hidden>
Date: Thu Aug 30 14:55:13 2012 -0400

    Address race condition from concurrent task state update

    task_state acts like a guard to avoid concurrent tasks to be
    scheduled. There might be two race conditions:

     1) two tasks are concurrently accepted by api, check task_state to be
     None and allow the tasks to be executed concurrently.

     2) one ordinary task is running, so that task_state is set. The delete
     task is accepted at API, and will "take over" and change task_state to
     DELETING. However the first task may continue to update task_state or
     set it to None as it finishes.

    This patch specifies current expected task_state when updating task_state.
    If unexpected state is met, abort the task without updating. Various
    compute tests are fixed accordingly to set the pre condition of the
    task state.

    Part of bug 1037372
    Change-Id: I5fdf0946c728a47febb56ad468043a828b2736c8

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-rc1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.