SecurityGroupExists error when booting multiple instances concurrently

Bug #1370782 reported by Roman Podoliaka
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Roman Podoliaka
Icehouse
Fix Released
Undecided
Unassigned

Bug Description

If the default security group doesn't exist for some particular tenant, booting of a few instances concurrently may lead to SecurityGroupExists error as one thread will win the race and create the security group, and others will fail.

This is easily reproduced by running Rally jobs in the gate.

Changed in nova:
assignee: nobody → Roman Podoliaka (rpodolyaka)
status: New → In Progress
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/122291

Joe Gordon (jogo)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
Matt Riedemann (mriedem) wrote :

This could be triggering heat race bug 1374021.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/122291
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b0e48fdaf7442a665d096bc934306d9517d921f0
Submitter: Jenkins
Branch: master

commit b0e48fdaf7442a665d096bc934306d9517d921f0
Author: Roman Podoliaka <email address hidden>
Date: Thu Sep 18 03:58:59 2014 +0300

    Fix SecurityGroupExists error when booting instances

    Make sure we properly handle the case when multiple threads check
    and create the default security group, if it hasn't existed yet.
    One thread will succeed and others will fail (we have a unique
    constraint in db to prevent a race condition), so just handle the
    exception to let instance boot proceed.

    Steps to reproduce:

    1) install OpenStack with Rally using DevStack:

        https://github.com/stackforge/rally/tree/master/contrib/devstack

    2) fetch the Rally task:

        http://boris-42.github.io/boot.yaml

    3) run it

        rally -v task start boot.yaml

    Closes-Bug: #1370782

    Change-Id: I414637a31b0b540fefa6c1314be6e973e2d4a178

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-rc1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/128593

Thierry Carrez (ttx)
Changed in nova:
milestone: juno-rc1 → 2014.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/128593
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ecc4688c1d619a559e53471fbb95652f3e9c97f
Submitter: Jenkins
Branch: stable/icehouse

commit 9ecc4688c1d619a559e53471fbb95652f3e9c97f
Author: Roman Podoliaka <email address hidden>
Date: Thu Sep 18 03:58:59 2014 +0300

    Fix SecurityGroupExists error when booting instances

    Make sure we properly handle the case when multiple threads check
    and create the default security group, if it hasn't existed yet.
    One thread will succeed and others will fail (we have a unique
    constraint in db to prevent a race condition), so just handle the
    exception to let instance boot proceed.

    Steps to reproduce:

    1) install OpenStack with Rally using DevStack:

        https://github.com/stackforge/rally/tree/master/contrib/devstack

    2) fetch the Rally task:

        http://boris-42.github.io/boot.yaml

    3) run it

        rally -v task start boot.yaml

    Closes-Bug: #1370782

    Conflicts:
            nova/tests/db/test_db_api.py

    Change-Id: I414637a31b0b540fefa6c1314be6e973e2d4a178
    (cherry picked from commit b0e48fdaf7442a665d096bc934306d9517d921f0)

tags: added: in-stable-icehouse
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.