vmware driver races to create instance images

Bug #1284312 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Shawn Hartsock
VMwareAPI-Team
In Progress
Critical
Shawn Hartsock

Bug Description

Change Ia0ebd674345734e7cfa31ccd400fdba93646c554 traded one race condition for another. By ignoring all mkdir() calls that would otherwise fail because an instance directory already exists, two nodes racing to create a single image will corrupt or lose data, or fail in a strange way. This call should fail in that case, but doesn't after the recent patch was merged:

https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vmops.py#L350

Tags: vmware
Changed in nova:
assignee: nobody → Shawn Hartsock (hartsock)
importance: Undecided → High
Matt Riedemann (mriedem)
tags: added: vmware
Changed in openstack-vmwareapi-team:
importance: Undecided → Critical
Revision history for this message
Matt Riedemann (mriedem) wrote :
Changed in openstack-vmwareapi-team:
assignee: nobody → Shawn Hartsock (hartsock)
Changed in nova:
milestone: none → icehouse-3
status: New → In Progress
Changed in openstack-vmwareapi-team:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/76071

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/75989
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2190e998c9667e4d4680d87a9d1643e25175dc8d
Submitter: Jenkins
Branch: master

commit 2190e998c9667e4d4680d87a9d1643e25175dc8d
Author: Dan Smith <email address hidden>
Date: Mon Feb 24 12:45:31 2014 -0800

    Revert "VMware: fix race for datastore directory existence"

    This reverts commit 743b74d59d3279478c512a0a559642b1d05284bc,
    which introduces a race condition when creating the initial
    instance image. Since _mkdir() no longer fails if the directory
    exists, two computes racing to create the same image will
    result in either corrupted data, lost data, or a strange
    failure later on.

    Change-Id: Ic646a01b2f96b1341af38b69c372875a95263bad
    Closes-bug: #1284312

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/76071
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a17d38b96c4b776de8938195f86953ff26f94c2f
Submitter: Jenkins
Branch: master

commit a17d38b96c4b776de8938195f86953ff26f94c2f
Author: Shawn Hartsock <email address hidden>
Date: Mon Feb 24 20:59:16 2014 -0500

    vmware: driver races to create instance images

    Addresses race condition that occurs when multiple processes
    arrive at a fresh Datastore with no vmware_base cache directory.

    Change-Id: If1bb6804e3668a56872a653a8ea44d0c7fc70311
    Co-Authored-by: Tracy Jones <email address hidden>
    closes-bug: #1284312

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.