Nova should not inspect cinder volume state during attach

Bug #1581230 reported by Walt Boring
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Ildiko Vancsa

Bug Description

The Nova API shouldn't be looking at the internal state of a Cinder volume to determine if it can attach. It should simply ask Cinder to reserve the volume. Cinder already has checks during reserve time for checking states.

Nova currently fetches a volume from Cinder, then checks to see if the state of the volume is in an appropriate state to attach, and then calls reserve. The problem is 2 fold.
1) Nova shouldn't care about the internal state management of Cinder Volumes. What happens if Cinder adds a new state that Nova doesn't know about?
2) By fetching a volume and checking, and then reserving, there is a race condition introduced. The state of the volume can change between the time the volume is fetched from Cinder and when reserve is called.

Nova should simply call reserve volume.

Tags: volumes
Changed in nova:
assignee: nobody → Walt Boring (walter-boring)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/315789

Changed in nova:
status: New → In Progress
tags: added: volumes
Changed in nova:
assignee: Walt Boring (walter-boring) → Ildiko Vancsa (ildiko-vancsa)
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/315789
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2e9959878bd7f2cc1f42c2b823aaff0163ef398c
Submitter: Jenkins
Branch: master

commit 2e9959878bd7f2cc1f42c2b823aaff0163ef398c
Author: Walter A. Boring IV <email address hidden>
Date: Thu May 12 14:59:29 2016 -0700

    Don't check cinder volume states during attach

    This patch changes how Nova interacts with volumes at attach time.
    Nova should rely upon Cinder's os-reserve to determine if the
    state of the volume is in a good/valid state for attaching.
    This fixes a race between when nova fetches the volume and calls
    os-reserve.

    This refactors the volume_api.check_attach a bit and adds a new
    check_availability_zone, which is still done on the Nova side.
    When Cinder's os-reserve supports passing in the availability
    zone, then this check can also be removed.

    This patch handles the volume attach API, which is not checked
    again in the compute manager. Future patches will handle other
    operations like boot from volume and swap volume.

    Partial-Bug: #1581230

    Change-Id: I5b069ba3480257c061541fc6c19e044c31417b5e

Changed in nova:
assignee: Ildiko Vancsa (ildiko-vancsa) → John Griffith (john-griffith)
Changed in nova:
assignee: John Griffith (john-griffith) → Ildiko Vancsa (ildiko-vancsa)
Changed in nova:
assignee: Ildiko Vancsa (ildiko-vancsa) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/335358
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=63805735c25a54ad1b9b97e05080c1a6153d8e22
Submitter: Jenkins
Branch: master

commit 63805735c25a54ad1b9b97e05080c1a6153d8e22
Author: Ildiko Vancsa <email address hidden>
Date: Tue Jun 28 16:05:53 2016 +0200

    Remove check_attach

    This patch finishes to remove the 'check_attach' call from Nova
    completely. As Cinder already performs the required checks as part
    of the 'reserve_volume' (os-reserve) call it is unnecessary to check the
    statemachine in Nova also and it can lead to race conditions.

    The missing 'reserve_volume' call is added to the BFV flow. In case of
    build failure the volume will be locked in 'attaching' state until the
    instance in ERROR state is cleaned up.

    We also check AZ for each volume attach operation which we haven't
    done for unshelve. A release note is added to enable 'cross_az_attach'
    in case the user does not care about AZ.

    The compute service version had to be bumped as the old computes still
    perform 'check_attach', which will fail when the API reserves the
    volume and the volume state moves to 'attaching'. If the computes
    are not new enough the old check will be called as opposed to
    'reserve_volume'.

    Closes-Bug: #1581230
    Change-Id: I3a3caa4c566ecc132aa2699f8c7e5987bbcc863a

Changed in nova:
status: In Progress → Fix Released
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Ildiko Vancsa (ildiko-vancsa)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.0.0b1

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.