storwize driver is not thread-safe

Bug #1274123 reported by Zoltan Arnold Nagy on 2014-01-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
High
Avishay Traeger

Bug Description

when launching multiple VMs at the same time (for example by selecting a higher than one instance count in the dashboard), the action will fail with various errors.

most likely the process goes as follows:
Thread1 + ... + ThreadN at the same time: check if the host is mapped -> it doesn't
Thread1: let's create the host!
...
ThreadN: let's create the host!
Thread1 succeeds
Thread2..N fails
Thread1 doesn't even finish the mapping _most of the time_

Zoltan Arnold Nagy (zoltan) wrote :

ok, same goes not for launching, but for deleting multiple VMs at the same time; it fails.

Avishay Traeger (avishay-il) wrote :

Correct, there is a race in both creating and deleting hosts.

tags: added: drivers storwize-svc
Changed in cinder:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Avishay Traeger (avishay-il)
milestone: none → icehouse-3

Fix proposed to branch: master
Review: https://review.openstack.org/72113

Changed in cinder:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/72113
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5ba26c8aa5b295a1202fbc6c01eb050ce3f3af3d
Submitter: Jenkins
Branch: master

commit 5ba26c8aa5b295a1202fbc6c01eb050ce3f3af3d
Author: Avishay Traeger <email address hidden>
Date: Sat Feb 8 22:59:58 2014 +0200

    Storwize/SVC: Fix races in host-related functions

    In the Storwize/SVC driver, initialize_connection checks if a host
    object exists on the storage, and if not, creates one. Similarly,
    terminate_connection checks if there are any more mappings for this
    host, and deletes the host object if there are none. Consequently, there
    exist races between two initialize_connection calls, two
    terminate_connection calls, or one and one.

    The easy solution here is to use locks. Because locks are files on the
    local machine, this implies that all cinder-volume processes managing a
    given Storwize/SVC contoller run on the same machine.

    Change-Id: I62cd992e7150e7d16f1a2b2be6ef61c64b638858
    Closes-Bug: #1274123

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-03-05
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in cinder:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers