Multibackend config issue in cinder-volume

[Heiko Krämer]

 Hi Guys,

I'm running in a strange config issue with cinder-volume service.
I try to use the multi backend feature in grizzly and the scheduler works fine but the volume service are not running correctly.
I can create/delete volumes but not attach.

My cinder.conf (abstract):

#### Backend Configuration
scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler
scheduler_host_manager=cinder.scheduler.host_manager.HostManager

enabled_backends=storage1,storage2
[storage1]
volume_group=nova-volumes
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_ISCSI
iscsi_helper=tgtadm

[storage2]
volume_group=nova-volumes
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_ISCSI
iscsi_helper=tgtadm

this section is on each host the same. If i try to attach an existing volume to an instance i'll get the following error on cinder-volume:

2013-04-16 17:18:13 AUDIT [cinder.service] Starting cinder-volume node (version 2013.1)
2013-04-16 17:18:13 INFO [cinder.volume.manager] Updating volume status
2013-04-16 17:18:13 INFO [cinder.volume.iscsi] Creating iscsi_target for: volume-b83ff42b-9a58-4bf9-8d95-945829d3ee9d
2013-04-16 17:18:13 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on 10.0.0.104:5672
2013-04-16 17:18:13 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on 10.0.0.104:5672
2013-04-16 17:18:14 INFO [cinder.volume.manager] Updating volume status
2013-04-16 17:18:14 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on 10.0.0.104:5672
2013-04-16 17:18:14 INFO [cinder.openstack.common.rpc.common] Connected to AMQP server on 10.0.0.104:5672
2013-04-16 17:18:26 ERROR [cinder.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 665, in initialize_connection
    return self.driver.initialize_connection(volume_ref, connector)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 336, in initialize_connection
    if self.configuration.iscsi_helper == 'lioadm':
  File "/usr/lib/python2.7/dist-packages/cinder/volume/configuration.py", line 83, in __getattr__
    return getattr(self.local_conf, value)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1708, in __getattr__
    return self._conf._get(name, self._group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1513, in _get
    value = self._substitute(self._do_get(name, group))
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1529, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 1629, in _get_opt_info
    raise NoSuchOptError(opt_name, group)
NoSuchOptError: no such option in group storage1: iscsi_helper

It's very strange the 'volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver' option should set the iscsi_helper=tgtadm per default.

P.S.: I've send to the mailinglist the same text but without any answers, so i think it's a bug or missconfiguration :/

Anyone have an idea or the same issue, otherwise i'll create a bug report.

Greetings from Berlin
Heiko

[Jérôme Gallard (jerome-gallard)]

for the records: http://<email address hidden>/msg22317.html

[Jérôme Gallard (jerome-gallard)]

For information, I managed to get the error with a fresh devstack install (multi-backend enabled).
However, it works if multi-backend is not enabled (ie, I'm able to attach volumes and use them).

In addition, when multi-backend is enabled, in my configuration, tempest test seems to be OK:
***
/opt/stack/tempest# nosetests -sv tempest.tests.volume.test_volumes_actions.py
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
tempest.tests.volume.test_volumes_actions.VolumesActionsTest.test_attach_detach_volume_to_instance[smoke] ... ok
tempest.tests.volume.test_volumes_actions.VolumesActionsTest.test_get_volume_attachment ... ok

----------------------------------------------------------------------
Ran 2 tests in 122.465s

OK
***

But... the test should failed? right?

[Jérôme Gallard (jerome-gallard)]

I don't know exactly why, but the issue seems to be linked with the following two lines :
cinder/volume/driver.py (l336 -337)

---
if self.configuration.iscsi_helper == 'lioadm':
       self.tgtadm.initialize_connection(volume, connector)
---

If I comment these two lines, the problem "is solved".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27155

[John Griffith (john-griffith)]

Hey Jerome,

Thanks for the detailed info, the problem here was improper use of configuration with multi-backend configuration. Should be good to go with the submitted patch.

[Avishay Traeger (avishay-il)]

I am out of the office until 05/12/2013.

For technical issues regarding the Storwize/SVC Cinder driver, please
contact: Jie Ping Wu <email address hidden>, Li Min Liu <email address hidden>,
Ronen Kat <email address hidden>
For all other issue, please contact my manager, Dalit Naor
<email address hidden>

Note: This is an automated response to your message "[Bug 1169928] Re:
Multibackend config issue in cinder-volume" sent on 19/04/2013 8:00:53.

This is the only notification you will receive while this person is away.

[Heiko Krämer (foexle)]

Hi John,

i've tested your bugfix and it's working now. I don't get any errors and can attach volumes to the instances.

Thanks and greetings
Heiko

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/27202

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27155
Committed: http://github.com/openstack/cinder/commit/168b7566a51807dfd42f873c810440472aed7171
Submitter: Jenkins
Branch: master

commit 168b7566a51807dfd42f873c810440472aed7171
Author: John Griffith <email address hidden>
Date: Thu Apr 11 18:42:59 2013 -0600

    Remove Flags usage from cinder.volume.driver

    cinder.volume.driver was using a mix of FLAGS and the
    new conf options for multi-backend. The result was failure
    of multi-backend configrations to set the iscsi_helper
    option.

    This updates the driver.py file to use the OSLO.cfg and fixes the
    check for iscsi_helper check in the initialize_connection call,
    which was undefined.

    Fixes bug: 1169928

    Change-Id: Ia1678893f32a9d08a695f055f59f7296e17d9d78

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27202
Committed: http://github.com/openstack/cinder/commit/6008be01d93c35240a37e8d589cbbf8b3d90471b
Submitter: Jenkins
Branch: stable/grizzly

commit 6008be01d93c35240a37e8d589cbbf8b3d90471b
Author: John Griffith <email address hidden>
Date: Thu Apr 11 18:42:59 2013 -0600

    Remove Flags usage from cinder.volume.driver

    cinder.volume.driver was using a mix of FLAGS and the
    new conf options for multi-backend. The result was failure
    of multi-backend configrations to set the iscsi_helper
    option.

    This updates the driver.py file to use the OSLO.cfg and fixes the
    check for iscsi_helper check in the initialize_connection call,
    which was undefined.

    Fixes bug: 1169928

    Change-Id: Ia1678893f32a9d08a695f055f59f7296e17d9d78
    (cherry picked from commit 168b7566a51807dfd42f873c810440472aed7171)

[Juan Pavlik (jjpavlik)]

Hi guys, i've the exact same problem with ubuntu 12.04+grizzly+cinder. But i don't have any new updates yet, is it going to be backported?

[John Griffith (john-griffith)]

Hi Juan,
This was backported and should show up when the distro updates hit.

[Juan Pavlik (jjpavlik)]

Awesome!!! I updated my server a couple of days ago and i have the changes! I'll test multibackend tomorrow, thanks guys!! great job!