In case of volume_use_multipath=True, Nova unable to fetch CONF.libvirt.volume_use_multipath value from nova.conf

Bug #1723928 reported by Vivek Soni
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Invalid
Undecided
Unassigned
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
os-brick
Invalid
Undecided
Unassigned

Bug Description

Issue :-
--------------
when we place 'volume_use_multipath=True' in nova.conf. while attaching the volume to an instance, 'connector' dictionary passed to cinder's initialize_connection() has multipath=False (i.e connector['multipath']=False)

Expected :-
--------------
This should be connector['multipath']=True since i have place 'volume_use_multipath=True'

connector
{'wwpns': [u'1000d4c9ef76a1d1', u'1000d4c9ef76a1d5'], 'wwnns': [u'2000d4c9ef76a1d1', u'2000d4c9ef76a1d5'], 'ip': '10.50.0.155', 'initiator': u'iqn.1993-08.org.debian:01:db6bf10a0db', 'platform': 'x86_64', 'host': 'cld6b11', 'do_local_attach': False, 'os_type': 'linux2', 'multipath': False}

Steps to reproduce :-
----------------------------
1) Place volume_use_multipath=True in nova.conf libvirt section
[libvirt]
live_migration_uri = qemu+ssh://stack@%s/system
cpu_mode = none
virt_type = kvm
volume_use_multipath = True

2) Create a lvm volume
3) Create a instance and try to attach.

Note :-
-----------------
This multipath functionality worked fine in Ocata. but from pike and current (queens) release this is not working as expected.

connector dictionary in ocata release :-
connector
{u'wwpns': [u'100038eaa73005a1', u'100038eaa73005a5'], u'wwnns': [u'200038eaa73005a1', u'200038eaa73005a5'], u'ip': u'10.50.128.110', u'initiator': u'iqn.1993-08.org.debian:01:d7f1c5d25e0', u'platform': u'x86_64', u'host': u'cld6b10', u'do_local_attach': False, u'os_type': u'linux2', u'multipath': True}

Revision history for this message
Matt Riedemann (mriedem) wrote :

Did you restart the nova-compute service after modifying nova.conf and before attaching the volume?

Revision history for this message
Matt Riedemann (mriedem) wrote :

Nova is just passing this value through to os-brick, so I don't really see anything wring with what Nova is doing here, so I've added os-brick to this bug report.

Please confirm if you are restarting the nova-compute service after modifying nova.conf and before attaching the volume.

Changed in nova:
status: New → Incomplete
Revision history for this message
Vivek Soni (viveksoni) wrote :

Yes i have restarted all the nova related services

Revision history for this message
Walt Boring (walter-boring) wrote :

This means the multipath daemon isn't running or hasn't been detected as running.
os-brick will look at the value coming nova, and also detect if the multipath daemon is running.
If and only if both of those are True, will os-brick return True in the connector.

I suspect this is just a misconfiguration on the user's part and/or the multipath daemon isn't actually installed/running.

Changed in os-brick:
status: New → Incomplete
Revision history for this message
Walt Boring (walter-boring) wrote :

You can see the code here where os-brick detects that multipathd is running.

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L154-L167

Please ensure multipathd is installed and running on your n-cpu hosts.

Revision history for this message
Vivek Soni (viveksoni) wrote :

multipathd is running :-
------------------------------
stack@cld6b11:/opt/stack/nova$ sudo multipathd show status
path checker states:
up 3

paths: 0
busy: False

Revision history for this message
Vivek Soni (viveksoni) wrote :
Vivek Soni (viveksoni)
Changed in nova:
status: Incomplete → New
Changed in os-brick:
status: Incomplete → New
Revision history for this message
Vivek Soni (viveksoni) wrote :

Walt, I have sent the n-cpu.log over mail

Revision history for this message
Vivek Soni (viveksoni) wrote :
Revision history for this message
Vivek Soni (viveksoni) wrote :

Nova.conf
--------------------------
[libvirt]
live_migration_uri = qemu+ssh://stack@%s/system
cpu_mode = none
virt_type = kvm
volume_use_multipath = True

Revision history for this message
William Durairaj (william-dur-sandanaraj) wrote :

While debugging a cinder nova volume-attach problem for multipathing on HPE 3PAR Driver, we found that the below source file does’nt honour the value of volume_use_multipath config value set in [libvirt] section of /etc/nova/nova.conf
Is this a known problem, since every other storage vendor driver will have the same issue.

In nova/virt/libvirt/driver.py

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1125

Value printed by CONF.libvirt.volume_use_multipath always returns False irrespective of whatever we set in section [libvirt] volume_use_multipath of /etc/nova/nova.conf
def get_volume_connector(self, instance):
        root_helper = utils.get_root_helper()
        return connector.get_connector_properties(
            root_helper, CONF.my_block_storage_ip,
            CONF.libvirt.volume_use_multipath,
            enforce_multipath=True,
            host=CONF.host)

Revision history for this message
William Durairaj (william-dur-sandanaraj) wrote :

Just for experimentation sake, I changed the default value to True in this line
https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L779

multipathing functionality was working as expected.

Matt Riedemann (mriedem)
tags: added: libvirt multipath volumes
Revision history for this message
Matt Riedemann (mriedem) wrote :

It doesn't make sense that the config option wouldn't be read from nova.conf and used in the volume drivers.

I need to see the config option dump from the start of the nova-compute service. When you configure nova.conf to set volume_use_multiattach=True and restart nova-compute, the logs should have a dump of the parsed config options in memory which the code will use. I need to see that output.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Which version of os-brick in ocata does this work with and which version of os-brick in pike does this not work with? Need to diff the history of changes in os-brick to see if there is something wrong there.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/532270

Matt Riedemann (mriedem)
Changed in cinder:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/532272

Revision history for this message
Matt Riedemann (mriedem) wrote :

Without the nova-compute part of the logs during the service startup where the config options are dumped from memory, I can't triage this. I've pushed up two patches that test how this option is being used in the libvirt driver code in nova and both show this working as expected.

Changed in nova:
status: New → Incomplete
Revision history for this message
Vivek Soni (viveksoni) wrote :

Thanks Matt for getting os-brick version

Just for others,
In Ocata, os-brick - 1.11.0 --- WORKING

In Pike, os-brick - 1.15.4 --- NOT WORKING

Revision history for this message
Vivek Soni (viveksoni) wrote :

stack@cld6b16:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| d81221cf-43c0-4d7b-b597-bc95e6f6d086 | available | vvk_vol | 1 | 3pariscsi | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
stack@cld6b16:~$ sudo systemctl restart devstack@n-*
stack@cld6b16:~$ nova volume-attach 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 d81221cf-43c0-4d7b-b597-bc95e6f6d086
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdb |
| id | d81221cf-43c0-4d7b-b597-bc95e6f6d086 |
| serverId | 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 |
| volumeId | d81221cf-43c0-4d7b-b597-bc95e6f6d086 |
+----------+--------------------------------------+
stack@cld6b16:~$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| d81221cf-43c0-4d7b-b597-bc95e6f6d086 | in-use | vvk_vol | 1 | 3pariscsi | false | 644f6590-a4f3-4ead-a9a1-509a09f3c1e7 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
stack@cld6b16:~$ sudo multipathd show status
path checker states:
up 4

paths: 0
busy: False
stack@cld6b16:~$

Revision history for this message
Vivek Soni (viveksoni) wrote :
Revision history for this message
Vivek Soni (viveksoni) wrote :
Revision history for this message
Vivek Soni (viveksoni) wrote :

Hi Matt,

the above n-cpu log also multipath = False, even if we add volume_use_multipath=True in nova.conf
-----------------------------
Jan 09 10:56:11 cld6b16 nova-compute[54607]: DEBUG oslo_service.service [None req-2bb753f4-bde2-4181-9cce-f3d68bc1d3e2 None None] libvirt.volume_use_multipath = False {{(pid=54607) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2888}}

Jan 09 10:56:16 cld6b16 nova-compute[54607]: DEBUG os_brick.utils [None req-6f1b3e07-57c8-4bca-af2f-fa2f4f038725 admin admin] ==> get_connector_properties: call u"{'execute': None, 'my_ip': '10.50.0.157', 'enforce_multipath': True, 'host': 'cld6b16', 'root_helper': 'sudo nova-rootwrap /etc/nova/rootwrap.conf', 'multipath': False}" {{(pid=54607) trace_logging_wrapper /usr/local/lib/python2.7/dist-packages/os_brick/utils.py:146}}

Revision history for this message
Matt Riedemann (mriedem) wrote :

Your nova-compute service is reading from /etc/nova/nova-cpu.conf, not nova.conf:

Jan 09 10:56:10 cld6b16 nova-compute[54607]: DEBUG oslo_service.service [None req-2bb753f4-bde2-4181-9cce-f3d68bc1d3e2 None None] config files: ['/etc/nova/nova-cpu.conf'] {{(pid=54607) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2870}}

So what's in nova-cpu.conf? Does it have the volume_use_multipath option set to True?

Revision history for this message
Vivek Soni (viveksoni) wrote :
Revision history for this message
Vivek Soni (viveksoni) wrote :

Hi Matt,

Initially in nova-cpu.conf, i have not set 'volume_use_multipath=True'.

After placing 'volume_use_multipath=True' in 'nova-cpu.conf', connector variable is populated with correct value from nova-cpu.conf.

Revision history for this message
Vivek Soni (viveksoni) wrote :

n-cpu.log file

Revision history for this message
Vivek Soni (viveksoni) wrote :

Thanks Matt

I tried to find the document/instruction, which states to place that flag in nova-cpu.conf instead of nova.conf, i was not able to find it.

May i request you to provide the document or links for the same, if not then i request to update the documents

Revision history for this message
Lee Yarwood (lyarwood) wrote :

The config-ref makes it clear that this needs to be configured on the individual compute hosts:

https://docs.openstack.org/nova/pike/configuration/config.html#libvirt
https://docs.openstack.org/nova/pike/configuration/config.html#libvirt.volume_use_multipath

Further as this is devstack you should be using local.conf to set this during the initial stack.sh run:

https://docs.openstack.org/devstack/latest/configuration.html#id3
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n51

Marking the bug as invalid given c#29

Changed in nova:
status: Incomplete → Invalid
Changed in os-brick:
status: New → Invalid
Revision history for this message
Matt Riedemann (mriedem) wrote :

I guess something could be added to the devstack docs, but I don't really know where someone should add that. But it also just takes a little bit of debugging to see which config file the actual service is running with and then check that you have the option set there.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/532270
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c315a9792379d79ad77d516d8babed6e0b6e58a8
Submitter: Zuul
Branch: master

commit c315a9792379d79ad77d516d8babed6e0b6e58a8
Author: Matt Riedemann <email address hidden>
Date: Tue Jan 9 12:26:28 2018 -0500

    libvirt: test to make sure volume_use_multipath is properly used

    There was no direct test for the libvirt.volume_use_multipath
    config option being used with the libvirt driver to get volume
    host connector properties from os-brick.

    This adds a unit test which sets the config option both ways before
    getting the connector properties and asserts the resulting value
    is the same as the config option value.

    Change-Id: If2b4cc477b4e7926bb2749db1eb4de2c6f694735
    Related-Bug: #1723928

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/532272
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2324146d8550fbf57d482f1238ad2294d61637cd
Submitter: Zuul
Branch: master

commit 2324146d8550fbf57d482f1238ad2294d61637cd
Author: Matt Riedemann <email address hidden>
Date: Tue Jan 9 12:43:34 2018 -0500

    libvirt: add tests to check multipath in iscsi/fc volume connectors

    This adds checks to the iscsi/fibrechannel volume driver connector
    tests to make sure that the resulting connector has a use_multipath
    value based on the libvirt.volume_use_multipath config option nova
    passes to os-brick.

    Change-Id: I82b303a405b8a2d51c2e659edf285fa1d1f3a73b
    Related-Bug: #1723928

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.