cinder-backup does not work with multi backend enabled

[Navneet]

The cinder-backup does not work if multi backend is enabled in cinder.conf for c-vol. The current design seems to depend on one backup service running for each volume driver backend, but only one backup service is spawned if c-bak is enabled. It does work with SimpleScheduler configured in cinder.conf and only one volume backend configured. It needs to spawn multiple cinder backup service per backend as per current design.

The problem seems to be more serious with the design of this feature. It spawns volume backend separately in c-vol and c-bak. There some backends which store a cache of provisioned volumes for performance reasons to quickly serve any request for that volume. As per the current design volume creation does update the cache in c-vol but it does not reflect in the driver instance spawned in c-bak service automatically. It seems to be a scalaility/performance problem for large number of backends configured as we are spawning exactly same number of backends in c-back as c-vol. I tend to incline toward rpc calls/messaging between c-vol and c-bak to do backups.

[John Griffith (john-griffith)]

I'll test this out, but could you please supply some more info. What exactly does "does not work" mean? Do you have logs, do you have description of the behavior that you encountered?

I'll look at it with multi-backends here shortly and see if I can reproduce, but future posts it would be great if you have additional detailed info.

[John Griffith (john-griffith)]

sighh... you're correct. The Backup service is checking against the global default volume-group only. And it would appear that it's all but completely broken in terms of multi-backend.

[John Griffith (john-griffith)]

The issue here is that there's currently no way for the backup manager to interpret the multiple volume hosts. It relies only on the base default config option which in the case of multi-backends will grab the unscoped entry for backend or the default (cinder-volumes)

2013-09-20 23:34:55.716 DEBUG amqp [-] Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2011 VMware, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'consumer_cancel_notify': True, u'publisher_confirms': True, u'basic.nack': True}, u'platform': u'Erlang/OTP', u'version': u'2.7.1'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US'] from (pid=16943) _start /usr/local/lib/python2.7/dist-packages/amqp/connection.py:706
2013-09-20 23:34:55.720 DEBUG amqp [-] Open OK! from (pid=16943) _open_ok /usr/local/lib/python2.7/dist-packages/amqp/connection.py:592
2013-09-20 23:34:55.720 DEBUG amqp [-] using channel_id: 1 from (pid=16943) __init__ /usr/local/lib/python2.7/dist-packages/amqp/channel.py:70
2013-09-20 23:34:55.723 DEBUG amqp [-] Channel open from (pid=16943) _open_ok /usr/local/lib/python2.7/dist-packages/amqp/channel.py:420
2013-09-20 23:34:55.724 INFO cinder.openstack.common.rpc.common [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Connected to AMQP server on localhost:5672
2013-09-20 23:34:55.725 DEBUG cinder.service [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Creating Consumer connection for Service cinder-backup from (pid=16943) start /opt/stack/cinder/cinder/service.py:371
2013-09-20 23:34:55.742 INFO cinder.backup.manager [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Starting volume driver LVMISCSIDriver (2.0.0)
2013-09-20 23:34:55.744 DEBUG cinder.openstack.common.processutils [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf vgs --noheadings -o name from (pid=16943) execute /opt/stack/cinder/cinder/openstack/common/processutils.py:142
2013-09-20 23:34:56.369 ERROR cinder.brick.local_dev.lvm [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Unable to locate Volume Group cinder-volumes
2013-09-20 23:34:56.373 ERROR cinder.backup.manager [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Error encountered during initialization of driver: LVMISCSIDriver
2013-09-20 23:34:56.374 ERROR cinder.backup.manager [req-edd476a4-9de2-435f-9ae1-c7562bf3b2a6 None None] Bad or unexpected response from the storage volume backend API: Volume Group cinder-volumes does not exist
2013-09-20 23:34:56.374 TRACE cinder.backup.manager Traceback (most recent call last):
2013-09-20 23:34:56.374 TRACE cinder.backup.manager File "/opt/stack/cinder/cinder/backup/manager.py", line 104, in init_host
2013-09-20 23:34:56.374 TRACE cinder.backup.manager self.driver.check_for_setup_error()
2013-09-20 23:34:56.374 TRACE cinder.backup.manager File "/opt/stack/cinder/cinder/volume/drivers/lvm.py", line 95, in check_for_setup_error
2013-09-20 23:34:56.374 TRACE cinder.backup.manager raise exception.VolumeBackendAPIException(data=message)
2013-09-2...

[Navneet (singn)]

Yeah...the volume host is the default for c-bak and does not exist corresponding to c-vol hosts. One dirty solution for this would be to start c-bak after c-vol and scan the c-vol hosts to spawn multiple services in c-bak.

Also as stated in the original report spawning separate copies of backends in c-bak and c-vol is a problem for some backends like us where we cache volume entries. A more robust solution would be to get the c-vol hosts and issue rpc requests to c-vol for serving requests. Please update your thoughts on this because the current design leaves our driver not completely supported with c-bak.

[Navneet (singn)]

One more use case to support for the rpc calls between c-bak and c-vol is that suppose we introduce many more new backends in running cinder with c-bak and c-vol. We will need to stop c-vol and restart it. But admins will not restart c-bak as they think the services are independent of each other and hence the new introduced backends will not be spawned in c-bak even if they start running in c-vol. This will cause backup to fail for new configured backends. In my view it requires that we have a centralized service for driver backends which can be contacted for different tasks.

[John Griffith (john-griffith)]

Navneet,
I agree, currently I'm looking at a solution for H, your proposal would be more appropriate for Icehouse, and I would like to see the backup service completely decoupled from the volume service.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/48460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/48460
Committed: http://github.com/openstack/cinder/commit/bd5c3f5a0e3449a4a384090ae4dba75c4cdcf4da
Submitter: Jenkins
Branch: master

commit bd5c3f5a0e3449a4a384090ae4dba75c4cdcf4da
Author: Edward Hope-Morley <email address hidden>
Date: Thu Sep 26 00:13:42 2013 +0100

    Fixes backup with multiple volume backend

    The backup service now keeps a dictionary of managers
    for each type of volume backend. If multi backend is
    not in use (and volume_type is None) the current
    volume_driver is used.

    Change-Id: I66e91f3ce2cffaf2e2d07edfddc08b1480e2a37c
    Fixes: bug 1228223