multipath not working with Storwize backend if CHAP enabled

[Zoltan Arnold Nagy]

if I try to attach a volume to a VM while having multipath enabled in nova and CHAP enabled in the storwize backend, it fails:

2014-09-09 11:37:14.038 22944 ERROR nova.virt.block_device [req-f271874a-9720-4779-96a8-01575641a939 a315717e20174b10a39db36b722325d6 76d25b1928e7407392a69735a894c7fc] [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Driver failed to attach volume c460f8b7-0f1d-4657-bdf7-e142ad34a132 at /dev/vdb
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Traceback (most recent call last):
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 239, in attach
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] device_type=self['device_type'], encryption=encryption)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1235, in attach_volume
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] disk_info)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1194, in volume_driver_method
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] return method(connection_info, *args, **kwargs)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] return f(*args, **kwargs)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 280, in connect_volume
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] check_exit_code=[0, 255])[0] \
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 579, in _run_iscsiadm_bare
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] check_exit_code=check_exit_code)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 165, in execute
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] return processutils.execute(*cmd, **kwargs)
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 193, in execute
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] cmd=' '.join(cmd))
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] ProcessExecutionError: Unexpected error while running command.
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.252:3260
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Exit code: 5
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Stdout: ''
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897] Stderr: 'iscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: Connection to Discovery Address 192.168.1.252 closed\niscsiadm: Login I/O error, failed to receive a PDU\niscsiadm: retrying discovery login to 192.168.1.252\niscsiadm: connection login retries (reopen_max) 5 exceeded\niscsiadm: Could not perform SendTargets discovery: encountered iSCSI login failure\n'
2014-09-09 11:37:14.038 22944 TRACE nova.virt.block_device [instance: 108a81d0-eeb5-49a8-b3eb-e593f44bf897]

the target clearly works and if I disable the CHAP secret in the driver it can attach the volume

[Jay Bryant (jsbryant)]

I have added Tao Bai from IBM to further investigate this.

[TaoBai (baitao2020)]

Jay, thanks for you to involve me

[Jay Bryant (jsbryant)]

Tao,

You are welcome. Trying to keep an eye out for these bugs so I can send them your way. :-)

[Geoff Willett (geoff-willett)]

We have this same issue in our environment. Works perfectly when multipath is disabled.

[TaoBai (baitao2020)]

I am trying to reproduce this issue with Kilo master branch.
But failed to reproduce it. Maybe something wrong with my configuration, or maybe already fixed.

I will continue to research it tomorrow.

Whatever, I just want to say I am working on it.

[Jay Bryant (jsbryant)]

Tao, I think this has already been fixed. I can't find the patch for it for some reason though. I will keep looking.

[Jay Bryant (jsbryant)]

Tao,

I found the change I was thinking of: https://review.openstack.org/#/c/129756/

commit c22038b9005070e51224f5057aac9f73cf4d0340
Author: Tomoki Sekiyama <email address hidden>
Date: Mon Oct 20 14:32:55 2014 -0400

    TgtAdm: Don't change CHAP username/password on live migration

    As tgtd doesn't update CHAP username/password while the initiator is
    connected, CHAP username/password must not be changed while a Nova
    instance are performing live-migration; otherwise the compute node
    which the instance migrates to cannot login to the volume and the
    migration process is aborted.

    This fixes TgtAdm implementation not to regenerate random
    username/password every time initialize_connection is called.
    Also, it enables CHAP auth in unit tests of TargetAdmin helpers.

    Change-Id: I48729a33fada62a7c8e4fe500b1e39533d898701
    Closes-Bug: #1383509

I am wondering if that change has made it so we don't have issues any more. I remember looking at this problem and thinking it looked like the credentials were getting messed up. Could you follow up and see if this appears plausible?

Thanks!

[TaoBai (baitao2020)]

Hi jay

Sorry miss leading you into wrong direction. I make wrong configuration in my Storage hardware. After I fixed the network configuration, I reproduced the issue. Sorry for the confusing.

Hi Jay, Geoff,Zoltan

After some debugging, here is my research result:
1. The error happened on " iscsiadm -m discovery -t sendtargets -p 192.168.1.252:3260" to discovery target iqn by "Nova"

2. Before step 1, storwize driver already create a "Chap enabled" host on storwize hardware.

3. After the "CHAP enabled" host created on storwize, "iscsiadm discovery" will need CHAP auth property, orelse the discovery will be rejected.
The properties are:
discovery.sendtargets.auth.authmethod
discovery.sendtargets.auth.username
discovery.sendtargets.auth.password

4.I went through the nova and Cinder code, did not find these CHAP discovery-auth property setup, so I think that's the reason why we discovery failed.
And I also set a breakpoint before nova iscsi discovery, and then update the auto properties, after that, success to discovery.

If I want to fix this bug, it will influence Nova and Cinder general code change.

I would propose other drivers verify this case before I modified it. I am not sure whether other drivers have the same issue (iscsi multipath and chap both enabled)or just IBM storwize.

[Sergey Gotliv (sgotliv)]

Tao,

I had an access to XtreamIO storage. Same thing happens there when I remove my credentials from iscsi daemon configuration file. Usually /etc/iscsi/iscsid.conf contains CHAP authentication properties for discovery command.

In my case, for example, it looks like:

# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
discovery.sendtargets.auth.username = sgotliv
discovery.sendtargets.auth.password = 123456789123

Technically this is a Nova bug because Nova runs discovery command in case it configured to work with the multipath.
Cinder provides all necessary information. Now the question is how do you want to update these parameters on the fly?

[TaoBai (baitao2020)]

Hi Sergey

Thanks for your confirmation that the issue not just for IBM storwize.

I went through Nova and Cinder code. I think the code change maybe done for both Cinder, and Nova. Cinder does not report the "Part 1" properties to nova but just the information for login "Part 2". (Even usually discovery and login use the same value for user and password value,just different keys)
example:
=====Not report by Cinder but needed by Nova======= Part 1
discovery.sendtargets.auth.authmethod
discovery.sendtargets.auth.username
discovery.sendtargets.auth.password
=============================================

Compared to :

=====Cinder already report for login================Part 2
node.session.auth.authmethod
node.session.auth.username
node.session.auth.password
=============================================

For me, there are two options.
Option 1:
 Cinder report additional discovery properties(part 1) , Nova use these information to do discovery. {both nova and cinder need to change}
Option 2:
Cinder no need change, but nova use "login properties" (part2)to discovery. But there is a risk that, even usually they are the same, but could be different.{only nova need to be change}

Personally, I prefer option 1.

[Sergey Gotliv (sgotliv)]

Tao,

Obviously, option #1 is a more complete solution!

The problem is that in order to implement it you probably will have to change all drivers or
make sure your code in Nova uses old properties when driver doesn't provide new ones.

[Geoff Willett (geoff-willett)]

HI Tao,

Please update this bug report with planned actions and timeline to fix. If this is a complex fix which might involve several vendors then I understand there will be some discussions as to the 'correct' solution, but I need to see some progress made.

Geoff.

[TaoBai (baitao2020)]

Hi Geoff
Sorry, I was interrupted by some other tasks and the holidays. I kept this ticket in my mind

I will start to do it next Monday after clean up some works on hand, I will think about the solution based on the discussion with Sergey

You can send me email: <email address hidden> if you have special or urgent need.

[TaoBai (baitao2020)]

After some thinking, I decided to choose option2. Because not Cinder driver distinguish Login password and discovery password(Please correct me if I am wrong)

The patch I uploaded is under discussion. https://review.openstack.org/#/c/147774/

[Sergey Gotliv (sgotliv)]

Tao,

Unfortunately, after reading other driver's code, I discovered that some of them generate
different credentials for login and discovery and do that on the fly which means you can't use
login credentials for discovery! So now I convinced that your option #1 is not just more
complete solution, but also the correct one.

[TaoBai (baitao2020)]

Hi Sergey

Thanks for your comments

I will change the code and use option1 posted above.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/148533

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148533
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=3f781f32d0aaa6adf9ef6c54ab6114de92b64917
Submitter: Jenkins
Branch: master

commit 3f781f32d0aaa6adf9ef6c54ab6114de92b64917
Author: TaoBai <email address hidden>
Date: Tue Jan 20 04:18:39 2015 -0800

    Failed to discovery when iscsi multipath and CHAP both enabled

    Storage server may be configured to protect target discovering phase with CHAP
    authentication, in this case existing discovery command will be failed in Nova
    when iscsi multipath enabled. Nova need these below discovery auth properties.
        "discovery.sendtargets.auth.authmethod",
        "discovery.sendtargets.auth.username",
        "discovery.sendtargets.auth.password"

    Cinder Storage driver need to send discovery auth properties to Nova in this
    case and the properties are:
       iscsi_properties['discovery_auth_method']
       iscsi_properties['discovery_auth_username']
       iscsi_properties['discovery_auth_password']

    This issue not just for IBM Storwize, but also other storage drivers who need
    CHAP authentication to do iscsi discover.
    The according nova change: https://review.openstack.org/#/c/148516/

    Closes-Bug: #1367189

    Change-Id: Ib528eed3a9fd5006c1649ef42233e1f943c38ab6

[TaoBai (baitao2020)]

https://review.openstack.org/#/c/148516/

The code change for nova has pending on +2 review but not merged for some days

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148516
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=45227bbbfd06d16e85e973e14ee8c30e267e8b42
Submitter: Jenkins
Branch: master

commit 45227bbbfd06d16e85e973e14ee8c30e267e8b42
Author: TaoBai <email address hidden>
Date: Thu Jan 15 22:49:49 2015 -0800

    Failed to discovery when iscsi multipath and CHAP both enabled

    Storage server may be configured to protect target discovering phase with CHAP
    authentication, in this case existing discovery command will be failed.
    The authentication properties are:
        "discovery.sendtargets.auth.authmethod",
        "discovery.sendtargets.auth.username",
        "discovery.sendtargets.auth.password"
    Cinder Storage driver need to send discovery auth properties in this case,
    and the properties are:
        iscsi_properties['discovery_auth_method']
        iscsi_properties['discovery_auth_username']
        iscsi_properties['discovery_auth_password']

    Change-Id: Ic70426d7d0fd8126879840f05341731ed92d6392
    Closes-Bug: #1367189

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155286

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/155289

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/155730

[OpenStack Infra (hudson-openstack)]

Change abandoned by Haïkel Guémar (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/155289
Reason: stable/icehouse is now in security maintenance

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155286
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c1493ab9cb00ada6503755186e0ac98c1a221bc1
Submitter: Jenkins
Branch: stable/juno

commit c1493ab9cb00ada6503755186e0ac98c1a221bc1
Author: TaoBai <email address hidden>
Date: Tue Jan 20 04:18:39 2015 -0800

    Failed to discovery when iscsi multipath and CHAP both enabled

    Storage server may be configured to protect target discovering phase with CHAP
    authentication, in this case existing discovery command will be failed in Nova
    when iscsi multipath enabled. Nova need these below discovery auth properties.
        "discovery.sendtargets.auth.authmethod",
        "discovery.sendtargets.auth.username",
        "discovery.sendtargets.auth.password"

    Cinder Storage driver need to send discovery auth properties to Nova in this
    case and the properties are:
       iscsi_properties['discovery_auth_method']
       iscsi_properties['discovery_auth_username']
       iscsi_properties['discovery_auth_password']

    This issue not just for IBM Storwize, but also other storage drivers who need
    CHAP authentication to do iscsi discover.
    The according nova change: https://review.openstack.org/#/c/148516/

    Closes-Bug: #1367189

    Change-Id: Ib528eed3a9fd5006c1649ef42233e1f943c38ab6
    (cherry picked from commit 3f781f32d0aaa6adf9ef6c54ab6114de92b64917)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155730
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aaf256acb5a34ee2d2f691092c0999a705767a8f
Submitter: Jenkins
Branch: stable/juno

commit aaf256acb5a34ee2d2f691092c0999a705767a8f
Author: TaoBai <email address hidden>
Date: Thu Jan 15 22:49:49 2015 -0800

    Failed to discovery when iscsi multipath and CHAP both enabled

    Storage server may be configured to protect target discovering phase with CHAP
    authentication, in this case existing discovery command will be failed.
    The authentication properties are:
        "discovery.sendtargets.auth.authmethod",
        "discovery.sendtargets.auth.username",
        "discovery.sendtargets.auth.password"
    Cinder Storage driver need to send discovery auth properties in this case,
    and the properties are:
        iscsi_properties['discovery_auth_method']
        iscsi_properties['discovery_auth_username']
        iscsi_properties['discovery_auth_password']

    (cherry picked from commit 45227bbbfd06d16e85e973e14ee8c30e267e8b42)

    Conflicts:
     nova/tests/virt/libvirt/test_volume.py

    Closes-Bug: #1367189
    Change-Id: Ic70426d7d0fd8126879840f05341731ed92d6392

[Walt Boring (walter-boring)]

Can you confirm that the latest os-brick solves this. I believe this patch has already added CHAP credentials to discovery here:

https://github.com/openstack/os-brick/commit/2ff5d8edca2edcb3da278fee8f5e8066e83b3d0e

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connector.py#L420

[Jay Bryant (jsbryant)]

Walt, I will get Yu Zhang on the Storewize test team to verify this.

Thanks!

[YuZhang (ivysdu)]

Verified, volume attach/detach works well with iSCSI multipath and CHAP enabled .

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/275175

[OpenStack Infra (hudson-openstack)]

Change abandoned by sahid (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/275175