XenServer attach second Cinder volume failed

Bug #1515672 reported by Sergey
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Bob Ball
Liberty
Fix Released
Medium
Kevin Bringard

Bug Description

With some Cinder backends (specifically those that provide multiple LUNs over the same IQN), XenServer cannot attach multiple disks to a VM.

Openstack kilo version nova-compute 2015.1.1-1.el7

Logs from nova-compute after attaching second volume to xenserver host:

2015-11-12 18:36:50.250 996 ERROR nova.virt.block_device [req-42ee1d90-2e83-41d3-aaec-3f928170b60d cb348d90ade04049b30e2d41a985e2d4 596095f4da8b4eddbfbbe1774e762bf4 - - -] [instance: 901b2692-e337-4358-80df-8408ce626352] Driver failed to attach volume a7ce4570-e35b-4db8-9502-8c33e790280a at /dev/xvdc
...
2015-11-12 18:36:50.250 996 TRACE nova.virt.block_device [instance: 901b2692-e337-4358-80df-8408ce626352] session.call_xenapi("SR.scan", sr_ref)
...
2015-11-12 18:36:50.250 996 TRACE nova.virt.block_device [instance: 901b2692-e337-4358-80df-8408ce626352] Failure: ['SR_BACKEND_FAILURE_40', '', 'The SR scan failed [opterr=[\'INTERNAL_ERROR\', \'Db_exn.Uniqueness_constraint_violation("VDI", "uuid", "70b87c59-e01a-be46-6f43-d927e3871669")\']]', '']

Reproduction steps

1. create first cinder volume on storage array - seen with a storwize 7000. (cinder create)
2. attach that volume to instance VM reside on xenserver host (first volume attached ok). (nova volume-attach)
3. create second cinder volume on storage array.
4. attach second cinder volume to instance VM on the same xenserver host as in step 2.
5. attach second volume failed

Expected result:

Second volume attached ok and gets available to VM instance

Bob Ball (bob-ball)
description: updated
Bob Ball (bob-ball)
tags: added: xenserver
summary: - Openstack cinder and XenServer attach second volume failed
+ XenServer attach second Cinder volume failed
Changed in nova:
status: New → Confirmed
Revision history for this message
Bob Ball (bob-ball) wrote :
description: updated
Bob Ball (bob-ball)
Changed in nova:
importance: Undecided → Medium
assignee: nobody → Bob Ball (bob-ball)
milestone: none → mitaka-1
Changed in nova:
status: Confirmed → In Progress
Matt Riedemann (mriedem)
tags: added: volumes
tags: added: kilo-backport-potential liberty-backport-potential
Changed in nova:
assignee: Bob Ball (bob-ball) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Bob Ball (bob-ball)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/244789
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5bd222e8d854ca7f03ee6936454ee57e0d6e1a78
Submitter: Jenkins
Branch: master

commit 5bd222e8d854ca7f03ee6936454ee57e0d6e1a78
Author: Bob Ball <email address hidden>
Date: Thu Nov 12 17:46:38 2015 +0000

    XenAPI: Cope with more Cinder backends

    Some Cinder backends (e.g. storwize 7000) expose multiple LUNs for a single
    SR connection from the XenServer host. Since the SR UUID was based on the
    'volume' ID, which is unique per volume, multiple SRs would be connecting
    to the same array when multiple volumes are exposed.

    This fixes the issue by making sure that we only connect to the array once
    for each (host,port,IQN) tuple, and we can re-use the SR for as many LUNs
    are presented.

    Change-Id: I6a1eb22fcfde8e3f5c069be54f5e7c91b0f2fec1
    Closes-Bug: 1515672

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b2

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/272695

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/277982

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/272695
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=deacf0efefcd9c0eaa99bf0b1836504d84c6800b
Submitter: Jenkins
Branch: stable/liberty

commit deacf0efefcd9c0eaa99bf0b1836504d84c6800b
Author: Bob Ball <email address hidden>
Date: Thu Nov 12 17:46:38 2015 +0000

    XenAPI: Cope with more Cinder backends

    Some Cinder backends (e.g. storwize 7000) expose multiple LUNs for a single
    SR connection from the XenServer host. Since the SR UUID was based on the
    'volume' ID, which is unique per volume, multiple SRs would be connecting
    to the same array when multiple volumes are exposed.

    This fixes the issue by making sure that we only connect to the array once
    for each (host,port,IQN) tuple, and we can re-use the SR for as many LUNs
    are presented.

    Change-Id: I6a1eb22fcfde8e3f5c069be54f5e7c91b0f2fec1
    Closes-Bug: 1515672
    (cherry picked from commit 5bd222e8d854ca7f03ee6936454ee57e0d6e1a78)

Matt Riedemann (mriedem)
tags: added: in-stable-liberty
removed: kilo-backport-potential liberty-backport-potential
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 12.0.2

This issue was fixed in the openstack/nova 12.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/kilo)

Change abandoned by Dave Walker (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/277982
Reason: Kilo is now approaching EOL. We are in freeze pending the final release and no freeze exception has been raised. Therefore I am abandoning this change, if it is required in the release - please restore and raise a request. Thanks

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.