Hyper-V unable to find mounted disks when attaching volume to instance on IBM Storwize V7K

[Lance Bragstad]

Steps to create:
1.) Create and instance, hosted on a Hyper-V system
2.) Create a new volume that is carved out of IBM Storwize V7K
3.) Confirm both are available/active
3.) Attach volume to instance
    $ nova volume-attach ldbragst-instance ldbragst-volume auto

The following stacktrace was pulled from the nova logs on the hyper-v node that hosts the instance.

[Hyper-V Volume Attach Issue]: Unable to find a mounted disk
http://paste.openstack.org/show/61387/

After this happens I can confirm that an iSCSI connection exists and is 'Connected' through the iSCSI Initiator. The iqn that it is connected to is the same iqn that is in the above stacktrace: iqn.1986-03.com.ibm:2145.ngp02-iscsi.node1

If I don't attempt to clean up anything on the Hyper-V node and don't 'Disconnect' the Discovered Targets (iqn.1986-03.com.ibm:2145.ngp02-iscsi.node1) and run the same exact 'nova attach-volume' command, I get a different error detailed below:

[Hyper-V Volume Attach Issue]: WMI Error Code
http://paste.openstack.org/show/61388/

Which I believe is being addressed in https://bugs.launchpad.net/nova/+bug/1221525 . I just wanted to note that it seems to be a side effect of the first issue.

Here, what I think is happening is that the device_number here https://github.com/openstack/nova/blob/master/nova/virt/hyperv/volumeops.py#L87 doesn't detect that a connection is established, so it tries to establish a connection, like the first run, by calling https://github.com/openstack/nova/blob/master/nova/virt/hyperv/volumeutilsv2.py#L45 and fails in this WMI call, https://github.com/openstack/nova/blob/master/nova/virt/hyperv/volumeutilsv2.py#L56

Here is some manual testing I did in a Python terminal on the compute node, specifically testing the case here https://github.com/openstack/nova/blob/master/nova/virt/hyperv/basevolumeutils.py#L122

http://paste.openstack.org/show/61389/

[Lance Bragstad (lbragstad)]

Hello Alex, outside of the linked traces and testing I did in Powershell, can you think of any additional things to try? We've been able to map a volume from the iSCSI target to the compute node running Hyper-V manually.

[Alessandro Pilotti (alexpilotti)]

Hi Lance, do you have updates on this bug?

Tx

[Jay Bryant (jsbryant)]

Just wanted to provide a public update on our progress here. We are working with IBM's PartnerWorld and Innovation Center to get Alex access to a v7k where this issue can be reproduced. Hopefully we will have more to update here soon.

[Jay Bryant (jsbryant)]

Just for documentation purposes, I would wanted to note here that I found the source of the problem in the storwize_svc driver and will be submitting a patch against the bug that this was duped against: bug #1221525 .

[Jay Bryant (jsbryant)]

Correction, as I looked at bug #1221525 they are two separate issues. I will be checking in the fix that the storwize_svc driver needs against this bug while I work on resolving the other bug.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/91931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91696
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=861e93a671f4ca4a7d8b08b05055bab9b9e141f8
Submitter: Jenkins
Branch: master

commit 861e93a671f4ca4a7d8b08b05055bab9b9e141f8
Author: Jay S. Bryant <email address hidden>
Date: Thu May 1 17:13:25 2014 -0500

    Ensure that lun_id is an int

    The map_vol_to_host function was treating the lun number
    as a string rather than an int. This was causing attempts
    to mount storwize_svc volumes to Hyper-V nodes to fail as they
    were checking an integer type against a unicode type which
    would fail.

    This change casts result_lun to an integer after the value has
    gone through the ssh injection attack check. This way Hyper-V
    is able to verify if the found LUN is the target LUN, enabling
    mount of storwize_svc volumes to Hyper-V.

    Change-Id: I96d179f2a58e5948c8ba6c9ae618daad7b8d4c86
    Closes-bug: 1269915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/91931
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d6829c180ea0056a166a3132832701f6b50f5679
Submitter: Jenkins
Branch: stable/icehouse

commit d6829c180ea0056a166a3132832701f6b50f5679
Author: Jay S. Bryant <email address hidden>
Date: Thu May 1 17:13:25 2014 -0500

    Ensure that lun_id is an int

    The map_vol_to_host function was treating the lun number
    as a string rather than an int. This was causing attempts
    to mount storwize_svc volumes to Hyper-V nodes to fail as they
    were checking an integer type against a unicode type which
    would fail.

    This change casts result_lun to an integer after the value has
    gone through the ssh injection attack check. This way Hyper-V
    is able to verify if the found LUN is the target LUN, enabling
    mount of storwize_svc volumes to Hyper-V.

    Change-Id: I96d179f2a58e5948c8ba6c9ae618daad7b8d4c86
    Closes-bug: 1269915
    (cherry picked from commit 861e93a671f4ca4a7d8b08b05055bab9b9e141f8)