Attach Volume fail : Cinder - ISCSI device symlinks under /dev/disk/by-path in hex.

[Vinay Prasad M]

As part of POC for an enterprise storage we have implemented an ISCSIDriver.
Volume operation works as expected.

Facing an issue with attach volume. Request your help.

1. If the volume lun id in backend storage is less than 255 attach volume works fine. Symlinks in /dev/disk/by-path is as below
lrwxrwxrwx 1 root root 9 Jun 18 15:18 ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-169 -> ../../sde
lrwxrwxrwx 1 root root 9 Aug 26 14:47 ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-172 -> ../../sdi

2. If volume lun id is more than 255, /dev/disk/by-path is an hexadecimal number and hence attach volume is failing with message Volume path not found. Symlinks in /dev/disk/by-path is as below [hexadecimal lun id according to SCSI standard ( REPORT LUNS)]
lrwxrwxrwx 1 root root 9 Aug 26 14:47 ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x016b000000000000 -> ../../sdh
lrwxrwxrwx 1 root root 9 Jun 18 15:18 ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x0200000000000000 -> ../../sdc

Please provide your suggession.

I would suggest cinder utility to have a check of both with normal lun-id in /dev/disk/by-path as well as hexadecimal lunid.

[John Griffith (john-griffith)]

Which release of OpenStack are you using? I'm not sure but there was some work in os-brick during liberty (handles initiator side) related to luns >255 for FC, I don't think this would be related but could be.

Also, the problem here I believe is a behavior in open-iscsi but it sounds like open-iscsi translates it to hex but the nova calls keep it as decimal?

[Vinay Prasad M (vpshelvankar)]

Thanks Griffith for the comment.

Currently I am implementing n testing driver with Kilo release. As you mentioned I do agree that open-iscsi translates lun-id to hex and Nova/cinder calls keep it as decimal while checking for device (symlinks)in /dev/disk/by-path.

I am not sure of where the compasion or check happens in this case.

[John Griffith (john-griffith)]

Try adding the check in your driver, when you build the provider info string, if it's >255 put the lun number in hex. That should work I think.

[Vinay Prasad M (vpshelvankar)]

We used the logic of converting the lun-id > 255 to hex in initialize_connection method of driver while passing target info to confirm the reason to fail. Attach volume works fine.

Is this workaround correct ??

Since the behavior is scci report luns standard and any array would face the same issue I do feel the logic has to go with os-brick instead of hex convert util in all drivers to check with hex if it fails to find device with lun-id > 255.

Also symlimks to device in /dev/disk/by-path is little unpredictable. I could not replicate the issue but have seen ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x0200000000000000-part1 couple of times. "-part1" appeared in the end and even this could make attach volume fail for the reason device not found.

/dev/disk/by-path isn't risky for attach volume ??

[Vinay Prasad M (vpshelvankar)]

Dear Griffith,

Any inputs and clarification around will be greatly appreciated. Thanks in
advance

Regards,
Vinay P
On Sep 13, 2015 12:40 AM, "John Griffith" <email address hidden>
wrote:

> Try adding the check in your driver, when you build the provider info
> string, if it's >255 put the lun number in hex. That should work I
> think.
>
> --
> You received this bug notification because you are a member of
> Toshibadrivers, which is subscribed to Cinder.
> https://bugs.launchpad.net/bugs/1493350
>
> Title:
> Attach Volume fail : Cinder - ISCSI device symlinks under /dev/disk
> /by-path in hex.
>
> Status in Cinder:
> New
>
> Bug description:
> As part of POC for an enterprise storage we have implemented an
> ISCSIDriver.
> Volume operation works as expected.
>
> Facing an issue with attach volume. Request your help.
>
> 1. If the volume lun id in backend storage is less than 255 attach
> volume works fine. Symlinks in /dev/disk/by-path is as below
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-169
> -> ../../sde
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-172
> -> ../../sdi
>
> 2. If volume lun id is more than 255, /dev/disk/by-path is an
> hexadecimal number and hence attach volume is failing with message Volume
> path not found. Symlinks in /dev/disk/by-path is as below [hexadecimal lun
> id according to SCSI standard ( REPORT LUNS)]
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x016b000000000000
> -> ../../sdh
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x0200000000000000
> -> ../../sdc
>
> Please provide your suggession.
>
> I would suggest cinder utility to have a check of both with normal
> lun-id in /dev/disk/by-path as well as hexadecimal lunid.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1493350/+subscriptions
>

[Vinay Prasad M (vpshelvankar)]

Dear Griffith,

Any inputs and clarification around will be greatly appreciated. Thanks in advance !!

[John Griffith (john-griffith)]

Hi Vinay, I was thinking of this bug and change here: https://bugs.launchpad.net/os-brick/+bug/1488233

[Vinay Prasad M (vpshelvankar)]

Thanks Griffith for the info on bug and the related change around FC.

Looking at the changes I am still not getting a clarity for the fix or the
exact reason.

Could you please assign this or request Stefan Amann (stefan-amann) who
fixed the above bug to have a look and comment on this.
On Sep 15, 2015 9:10 AM, "John Griffith" <email address hidden> wrote:

> Hi Vinay, I was thinking of this bug and change here:
> https://bugs.launchpad.net/os-brick/+bug/1488233
>
> --
> You received this bug notification because you are a member of
> Toshibadrivers, which is subscribed to Cinder.
> https://bugs.launchpad.net/bugs/1493350
>
> Title:
> Attach Volume fail : Cinder - ISCSI device symlinks under /dev/disk
> /by-path in hex.
>
> Status in Cinder:
> New
>
> Bug description:
> As part of POC for an enterprise storage we have implemented an
> ISCSIDriver.
> Volume operation works as expected.
>
> Facing an issue with attach volume. Request your help.
>
> 1. If the volume lun id in backend storage is less than 255 attach
> volume works fine. Symlinks in /dev/disk/by-path is as below
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-169
> -> ../../sde
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-172
> -> ../../sdi
>
> 2. If volume lun id is more than 255, /dev/disk/by-path is an
> hexadecimal number and hence attach volume is failing with message Volume
> path not found. Symlinks in /dev/disk/by-path is as below [hexadecimal lun
> id according to SCSI standard ( REPORT LUNS)]
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x016b000000000000
> -> ../../sdh
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x0200000000000000
> -> ../../sdc
>
> Please provide your suggession.
>
> I would suggest cinder utility to have a check of both with normal
> lun-id in /dev/disk/by-path as well as hexadecimal lunid.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1493350/+subscriptions
>

[Vinay Prasad M (vpshelvankar)]

Hi Griffith,

I am still unable to get a fix for the bug.
Can this be assigned to cinder-bug-team group ?

[Sylvain Bauza (sylvain-bauza)]

I'm really not sure what needs to be done in Nova to fix that problem. Feel free to reopen the bug to New if you come up with a described description of any Nova-related issue coming in.

[Walt Boring (walter-boring)]

What storage backend are you using? Are you sure your backend isn't exporting the LUN id's via hex?

[Walt Boring (walter-boring)]

So I did a test with my 3PAR iSCSI array and was able to reproduce this. The 3PAR is exported the lun id as 257

http://paste.openstack.org/show/481102/

...

lrwxrwxrwx 1 root root 10 Dec 7 16:17 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-248 -> ../../sdip
lrwxrwxrwx 1 root root 10 Dec 7 16:17 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-250 -> ../../sdir
lrwxrwxrwx 1 root root 10 Dec 7 16:17 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-251 -> ../../sdis
lrwxrwxrwx 1 root root 10 Dec 7 16:17 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-252 -> ../../sdit
lrwxrwxrwx 1 root root 10 Dec 7 16:17 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-253 -> ../../sdiu
lrwxrwxrwx 1 root root 10 Dec 7 16:19 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-255 -> ../../sdiv
lrwxrwxrwx 1 root root 9 Dec 7 16:20 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 10 Dec 7 16:20 ip-10.52.1.11:3260-iscsi-iqn.2000-05.com.3pardata:20810002ac00383d-lun-0x0100000000000000 -> ../../sdiw

You can see the output here getting odd. Could be an artifact of the sg_info tool itself though....
sg_info -d -l /dev/sdiw output:

    00fd000000000000 [253]
      Peripheral device addressing: lun=253
    00fe000000000000 [254]
      Peripheral device addressing: lun=254
    00ff000000000000 [255]
      Peripheral device addressing: lun=255
    0101000000000000 [257]
      Peripheral device addressing: bus_id=1, target=1
      >>Second level addressing:
        Peripheral device addressing: lun=0

[Vinay Prasad M (vpshelvankar)]

Thanks Walt for your reply.

We are using Toshiba Storage Array as our backend. We are sure that the backend is not exporting the LUN id's via hex.

Thanks again for reproducing and confirming with 3PAR iSCSI array too. We are not able to identify the generic fix. However in our Toshiba Driver we have added logic to convert lun-id to hex if > 255. This might not be the right approach. The bug has to be fixed in general. Can you please have a look and help us providing the fix.

[Dilip S M (itzdilip)]

Hi Walt ,

which version of openstack did you test?

are you able to make out where it is suppose to be get fixed?

Thanks
Dilip

[Walt Boring (walter-boring)]

The proper fix to this is going to be in os-brick. The stock udev rules is creating the paths, so we can't change that. We need to account for it in each of the connectors that look for those paths and using LUN ids. Instead of simply using the LUN id that comes into connect_volume, we'll have to translate and make that available for all the connectors.
basically if LUN >= 256 then use the hex, else use LUN id.

[John Griffith (john-griffith)]

@Walt
+1 just need to add a check/translate to hex on the iscsiadm calls

[Walt Boring (walter-boring)]

Vinay, do you have a patch up for this yet? You can't set the status to In Progress without having a patch up.

[John Griffith (john-griffith)]

@Vinay,
Are you working on this still? If not let me know, and reassign and I'll submit a patch.

[Vinay Prasad M (vpshelvankar)]

@Walt and John,

I have fixed the bug and have a patch up. I am unable to git-review as SSH is blocked in my org. Trying with https config but failing as below -
stack@ubuntu:~/os-brick$ git review -s
Problems encountered installing commit-msg hook
The following command failed with exit code 1
    "scp vpshelvankar:<email address hidden>:hooks/commit-msg .git/hooks/commit-msg"
-----------------------
.git/hooks/commit-msg: No such file

Will do it by today.

Regards
Vinay P

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264578

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vinay Prasad M (<email address hidden>) on branch: master
Review: https://review.openstack.org/264578
Reason: Indentation issues since i created patch from Windows machine. Abandoning current patch and will upload new

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264671

[Vinay Prasad M (vpshelvankar)]

@John,

Have already mailed to Kendall regarding this. He is taking it further.

-VinayP
On Feb 9, 2016 8:20 PM, "Kendall Nelson" <email address hidden> wrote:

> ** Changed in: os-brick
> Assignee: Vinay Prasad M (vpshelvankar) => Kendall Nelson (kjnelson)
>
> --
> You received this bug notification because you are a member of
> Toshibadrivers, which is subscribed to Cinder.
> https://bugs.launchpad.net/bugs/1493350
>
> Title:
> Attach Volume fail : Cinder - ISCSI device symlinks under /dev/disk
> /by-path in hex.
>
> Status in Cinder:
> Confirmed
> Status in OpenStack Compute (nova):
> Opinion
> Status in os-brick:
> In Progress
>
> Bug description:
> As part of POC for an enterprise storage we have implemented an
> ISCSIDriver.
> Volume operation works as expected.
>
> Facing an issue with attach volume. Request your help.
>
> 1. If the volume lun id in backend storage is less than 255 attach
> volume works fine. Symlinks in /dev/disk/by-path is as below
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-169
> -> ../../sde
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xyz:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-172
> -> ../../sdi
>
> 2. If volume lun id is more than 255, /dev/disk/by-path is an
> hexadecimal number and hence attach volume is failing with message Volume
> path not found. Symlinks in /dev/disk/by-path is as below [hexadecimal lun
> id according to SCSI standard ( REPORT LUNS)]
> lrwxrwxrwx 1 root root 9 Aug 26 14:47
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x016b000000000000
> -> ../../sdh
> lrwxrwxrwx 1 root root 9 Jun 18 15:18
> ip-192.168.5.100:3260-iscsi-iqn.1989-10.jp.co.xxx:storage.ttsp.2ec7fdda.ff5c4a16.0-lun-0x0200000000000000
> -> ../../sdc
>
> Please provide your suggession.
>
> I would suggest cinder utility to have a check of both with normal
> lun-id in /dev/disk/by-path as well as hexadecimal lunid.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cinder/+bug/1493350/+subscriptions
>

[Walt Boring (walter-boring)]

For what it's worth, here is the udev code that does the conversion and formatting.

http://bazaar.launchpad.net/~vcs-imports/udev/trunk/view/head:/src/udev-builtin-path_id.c#L67

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277606
Committed: https://git.openstack.org/cgit/openstack/os-brick/commit/?id=82cdb40f870c958da6ab6ef447b35be2eaec5c8d
Submitter: Jenkins
Branch: master

commit 82cdb40f870c958da6ab6ef447b35be2eaec5c8d
Author: Jenkins <email address hidden>
Date: Thu Feb 11 17:22:19 2016 +0000

    Lun id's > 255 should be converted to hex

    This patch addresses the issue where lun id's values are larger
    than 255 and are being kept as integers. This causes the volume and
    search paths to be malformed and volumes can't be found. This patch
    adds two functions to linuxscsi.py to process the lun id's; they can
    process both a single lun id and a list of them. If a lun id has a
    value larger than 255 it is converted to hex. This patch also modifies
    the necessary unit tests and adds ones to cover the new main function.

    Change-Id: Ib0b2f239a8152275de9ea66fa99a286dfbe53d57
    Closes-bug: #1493350

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/264671
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.