In case of volume_use_multipath=True, Nova unable to fetch CONF.libvirt.volume_use_multipath value from nova.conf

[Vivek Soni]

Issue :-
--------------
when we place 'volume_use_multipath=True' in nova.conf. while attaching the volume to an instance, 'connector' dictionary passed to cinder's initialize_connection() has multipath=False (i.e connector['multipath']=False)

Expected :-
--------------
This should be connector['multipath']=True since i have place 'volume_use_multipath=True'

connector
{'wwpns': [u'1000d4c9ef76a1d1', u'1000d4c9ef76a1d5'], 'wwnns': [u'2000d4c9ef76a1d1', u'2000d4c9ef76a1d5'], 'ip': '10.50.0.155', 'initiator': u'iqn.1993-08.org.debian:01:db6bf10a0db', 'platform': 'x86_64', 'host': 'cld6b11', 'do_local_attach': False, 'os_type': 'linux2', 'multipath': False}

Steps to reproduce :-
----------------------------
1) Place volume_use_multipath=True in nova.conf libvirt section
[libvirt]
live_migration_uri = qemu+ssh://stack@%s/system
cpu_mode = none
virt_type = kvm
volume_use_multipath = True

2) Create a lvm volume
3) Create a instance and try to attach.

Note :-
-----------------
This multipath functionality worked fine in Ocata. but from pike and current (queens) release this is not working as expected.

connector dictionary in ocata release :-
connector
{u'wwpns': [u'100038eaa73005a1', u'100038eaa73005a5'], u'wwnns': [u'200038eaa73005a1', u'200038eaa73005a5'], u'ip': u'10.50.128.110', u'initiator': u'iqn.1993-08.org.debian:01:d7f1c5d25e0', u'platform': u'x86_64', u'host': u'cld6b10', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': True}

[Matt Riedemann (mriedem)]

Did you restart the nova-compute service after modifying nova.conf and before attaching the volume?

[Matt Riedemann (mriedem)]

Nova is just passing this value through to os-brick, so I don't really see anything wring with what Nova is doing here, so I've added os-brick to this bug report.

Please confirm if you are restarting the nova-compute service after modifying nova.conf and before attaching the volume.

[Vivek Soni (viveksoni)]

Yes i have restarted all the nova related services

[Walt Boring (walter-boring)]

This means the multipath daemon isn't running or hasn't been detected as running.
os-brick will look at the value coming nova, and also detect if the multipath daemon is running.
If and only if both of those are True, will os-brick return True in the connector.

I suspect this is just a misconfiguration on the user's part and/or the multipath daemon isn't actually installed/running.

[Walt Boring (walter-boring)]

You can see the code here where os-brick detects that multipathd is running.

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L154-L167

Please ensure multipathd is installed and running on your n-cpu hosts.

[Vivek Soni (viveksoni)]

multipathd is running :-
------------------------------
stack@cld6b11:/opt/stack/nova$ sudo multipathd show status
path checker states:
up 3

paths: 0
busy: False

[Vivek Soni (viveksoni)]

Walt, I have sent the n-cpu.log over mail

[Vivek Soni (viveksoni)]

Nova.conf
--------------------------
[libvirt]
live_migration_uri = qemu+ssh://stack@%s/system
cpu_mode = none
virt_type = kvm
volume_use_multipath = True

[William Durairaj (william-dur-sandanaraj)]

While debugging a cinder nova volume-attach problem for multipathing on HPE 3PAR Driver, we found that the below source file does’nt honour the value of volume_use_multipath config value set in [libvirt] section of /etc/nova/nova.conf
Is this a known problem, since every other storage vendor driver will have the same issue.

In nova/virt/libvirt/driver.py

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1125

Value printed by CONF.libvirt.volume_use_multipath always returns False irrespective of whatever we set in section [libvirt] volume_use_multipath of /etc/nova/nova.conf
def get_volume_connector(self, instance):
        root_helper = utils.get_root_helper()
        return connector.get_connector_properties(
            root_helper, CONF.my_block_storage_ip,
            CONF.libvirt.volume_use_multipath,
            enforce_multipath=True,
            host=CONF.host)

[William Durairaj (william-dur-sandanaraj)]

Just for experimentation sake, I changed the default value to True in this line
https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L779

multipathing functionality was working as expected.

[Matt Riedemann (mriedem)]

It doesn't make sense that the config option wouldn't be read from nova.conf and used in the volume drivers.

I need to see the config option dump from the start of the nova-compute service. When you configure nova.conf to set volume_use_multiattach=True and restart nova-compute, the logs should have a dump of the parsed config options in memory which the code will use. I need to see that output.

[Matt Riedemann (mriedem)]

Which version of os-brick in ocata does this work with and which version of os-brick in pike does this not work with? Need to diff the history of changes in os-brick to see if there is something wrong there.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532270

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532272

[Matt Riedemann (mriedem)]

Without the nova-compute part of the logs during the service startup where the config options are dumped from memory, I can't triage this. I've pushed up two patches that test how this option is being used in the libvirt driver code in nova and both show this working as expected.

[Vivek Soni (viveksoni)]

Thanks Matt for getting os-brick version

Just for others,
In Ocata, os-brick - 1.11.0 --- WORKING

In Pike, os-brick - 1.15.4 --- NOT WORKING

[Vivek Soni (viveksoni)]

stack@cld6b16:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| d81221cf-43c0-4d7b-b597-bc95e6f6d086 | available | vvk_vol | 1 | 3pariscsi | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
stack@cld6b16:~$ sudo systemctl restart devstack@n-*
stack@cld6b16:~$ nova volume-attach 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 d81221cf-43c0-4d7b-b597-bc95e6f6d086
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d81221cf-43c0-4d7b-b597-bc95e6f6d086 |
| serverId | 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 |
| volumeId | d81221cf-43c0-4d7b-b597-bc95e6f6d086 |
+----------+--------------------------------------+
stack@cld6b16:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| d81221cf-43c0-4d7b-b597-bc95e6f6d086 | in-use | vvk_vol | 1 | 3pariscsi | false | 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
stack@cld6b16:~$ sudo multipathd show status
path checker states:
up 4

paths: 0
busy: False
stack@cld6b16:~$

[Vivek Soni (viveksoni)]

Hi Matt,

the above n-cpu log also multipath = False, even if we add volume_use_multipath=True in nova.conf
-----------------------------
Jan 09 10:56:11 cld6b16 nova-compute[54607]: DEBUG oslo_service.service [None req-2bb753f4-bde2-4181-9cce-f3d68bc1d3e2 None None] libvirt.volume_use_multipath = False {{(pid=54607) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2888}}

Jan 09 10:56:16 cld6b16 nova-compute[54607]: DEBUG os_brick.utils [None req-6f1b3e07-57c8-4bca-af2f-fa2f4f038725 admin admin] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '10.50.0.157', 'enforce_multipath': True, 'host': 'cld6b16', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" {{(pid=54607) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:146}}

[Matt Riedemann (mriedem)]

Your nova-compute service is reading from /etc/nova/nova-cpu.conf, not nova.conf:

Jan 09 10:56:10 cld6b16 nova-compute[54607]: DEBUG oslo_service.service [None req-2bb753f4-bde2-4181-9cce-f3d68bc1d3e2 None None] config files: ['/etc/nova/nova-cpu.conf'] {{(pid=54607) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2870}}

So what's in nova-cpu.conf? Does it have the volume_use_multipath option set to True?

[Vivek Soni (viveksoni)]

Hi Matt,

Initially in nova-cpu.conf, i have not set 'volume_use_multipath=True'.

After placing 'volume_use_multipath=True' in 'nova-cpu.conf', connector variable is populated with correct value from nova-cpu.conf.

[Vivek Soni (viveksoni)]

n-cpu.log file

[Vivek Soni (viveksoni)]

Thanks Matt

I tried to find the document/instruction, which states to place that flag in nova-cpu.conf instead of nova.conf, i was not able to find it.

May i request you to provide the document or links for the same, if not then i request to update the documents

[Lee Yarwood (lyarwood)]

The config-ref makes it clear that this needs to be configured on the individual compute hosts:

https://docs.openstack.org/nova/pike/configuration/config.html#libvirt
https://docs.openstack.org/nova/pike/configuration/config.html#libvirt.volume_use_multipath

Further as this is devstack you should be using local.conf to set this during the initial stack.sh run:

https://docs.openstack.org/devstack/latest/configuration.html#id3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n51

Marking the bug as invalid given c#29

[Matt Riedemann (mriedem)]

I guess something could be added to the devstack docs, but I don't really know where someone should add that. But it also just takes a little bit of debugging to see which config file the actual service is running with and then check that you have the option set there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532270
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c315a9792379d79ad77d516d8babed6e0b6e58a8
Submitter: Zuul
Branch: master

commit c315a9792379d79ad77d516d8babed6e0b6e58a8
Author: Matt Riedemann <email address hidden>
Date: Tue Jan 9 12:26:28 2018 -0500

    libvirt: test to make sure volume_use_multipath is properly used

    There was no direct test for the libvirt.volume_use_multipath
    config option being used with the libvirt driver to get volume
    host connector properties from os-brick.

    This adds a unit test which sets the config option both ways before
    getting the connector properties and asserts the resulting value
    is the same as the config option value.

    Change-Id: If2b4cc477b4e7926bb2749db1eb4de2c6f694735
    Related-Bug: #1723928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532272
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2324146d8550fbf57d482f1238ad2294d61637cd
Submitter: Zuul
Branch: master

commit 2324146d8550fbf57d482f1238ad2294d61637cd
Author: Matt Riedemann <email address hidden>
Date: Tue Jan 9 12:43:34 2018 -0500

    libvirt: add tests to check multipath in iscsi/fc volume connectors

    This adds checks to the iscsi/fibrechannel volume driver connector
    tests to make sure that the resulting connector has a use_multipath
    value based on the libvirt.volume_use_multipath config option nova
    passes to os-brick.

    Change-Id: I82b303a405b8a2d51c2e659edf285fa1d1f3a73b
    Related-Bug: #1723928