Fuel for OpenStack

Cinder backup service can not create volume backup with ceph enabled

Bug #1441109 reported by Kyrylo Romanenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Bartłomiej Piotrowski
Fuel for OpenStack 6.1

Bug Description

Cinder backup service can not create volume backup

Steps to reproduce:
1. Checked that cinder-backup service is running:
root@node-1:~# service cinder-backup status
cinder-backup start/running, process 32491
root@node-1:~# cinder service-list
+------------------+-------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | rbd:volumes | nova | enabled | up | 2015-04-07T10:44:26.000000 | None |
| cinder-scheduler | rbd:volumes | nova | enabled | up | 2015-04-07T10:44:28.000000 | None |
| cinder-volume | rbd:volumes | nova | enabled | up | 2015-04-07T10:44:28.000000 | None |
+------------------+-------------+------+---------+-------+----------------------------+-----------------+

2. Created a Volume
# cinder create 1 --display-name VOLUME1
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-04-07T10:31:17.963122 |
| display_description | None |
| display_name | VOLUME1 |
| encrypted | False |
| id | 6ad0216d-9e34-4db9-8837-92f3feca7073 |
| metadata | {} |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+---------------------+--------------------------------------+

3. Check that volume available:

 cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 6ad0216d-9e34-4db9-8837-92f3feca7073 | available | VOLUME1 | 1 | None | false | |

4. Create volume backup:
# cinder backup-create 6ad0216d-9e34-4db9-8837-92f3feca7073
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | aad20643-07a7-4d8c-834d-c2741189848f |
| name | None |
| volume_id | 6ad0216d-9e34-4db9-8837-92f3feca7073 |
+-----------+--------------------------------------+

5. Note error in backup list
 # cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+-----------+
| aad20643-07a7-4d8c-834d-c2741189848f | 6ad0216d-9e34-4db9-8837-92f3feca7073 | error | None | 1 | None | None |

6. Inspect backup details:
# cinder backup-show aad20643-07a7-4d8c-834d-c2741189848f
+-------------------+--------------------------------------+
| Property | Value |
+-------------------+--------------------------------------+
| availability_zone | None |
| container | None |
| created_at | 2015-04-07T10:37:52.000000 |
| description | None |
| fail_reason | Volume driver not ready. |
| id | aad20643-07a7-4d8c-834d-c2741189848f |
| name | None |
| object_count | None |
| size | 1 |
| status | error |
| volume_id | 6ad0216d-9e34-4db9-8837-92f3feca7073 |
+-------------------+--------------------------------------+

Note: | fail_reason | Volume driver not ready. |

6. Inspect error in cinder-backup log.
/var/log/cinder/cinder-backup.log

2015-04-07 10:50:06.986 32491 INFO cinder.backup.manager [req-c1b7c11c-77c5-40b9-ac28-4b9dc166c342 e63b889425af4c9c804adea1188d5b9c 3445d95ceb3b4f5f998b542bbe33ebaa - - -] Create backup started, backup: 5545bd2a-ff5f-48c2-a630-0077a84577da volume: 6ad0216d-9e34-4db9-8837-92f3feca7073.
2015-04-07 10:50:06.990 32491 INFO cinder.backup.manager [req-c1b7c11c-77c5-40b9-ac28-4b9dc166c342 e63b889425af4c9c804adea1188d5b9c 3445d95ceb3b4f5f998b542bbe33ebaa - - -] Backend not found in hostname (rbd:volumes) so using default.
2015-04-07 10:50:07.008 32491 INFO cinder.backup.manager [req-c1b7c11c-77c5-40b9-ac28-4b9dc166c342 e63b889425af4c9c804adea1188d5b9c 3445d95ceb3b4f5f998b542bbe33ebaa - - -] Backend not found in hostname (None) so using default.
2015-04-07 10:50:07.014 32491 ERROR cinder.utils [req-c1b7c11c-77c5-40b9-ac28-4b9dc166c342 e63b889425af4c9c804adea1188d5b9c 3445d95ceb3b4f5f998b542bbe33ebaa - - -] Volume driver RBDDriver not initialized
2015-04-07 10:50:07.164 32491 ERROR oslo.messaging.rpc.dispatcher [req-c1b7c11c-77c5-40b9-ac28-4b9dc166c342 e63b889425af4c9c804adea1188d5b9c 3445d95ceb3b4f5f998b542bbe33ebaa - - -] Exception during message handling: Volume driver not ready.
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 180, in _dispatch
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 126, in _do_dispatch
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 288, in create_backup
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher 'fail_reason': unicode(err)})
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 82, in __exit__
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 277, in create_backup
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher utils.require_driver_initialized(self.driver)
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 619, in require_driver_initialized
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher raise exception.DriverNotInitialized()
2015-04-07 10:50:07.164 32491 TRACE oslo.messaging.rpc.dispatcher DriverNotInitialized: Volume driver not ready.

Environment:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "282"
  build_id: "2015-04-06_08-49-30"
  nailgun_sha: "a2dc04af9c5838a990adeab81789d4c090943701"
  python-fuelclient_sha: "9ad42671130deb2cc81135fb453ef0793883be2c"
  astute_sha: "240bfefe0a0b5af21b79854d0c96d8a3ee0fd885"
  fuellib_sha: "601cc8e1dccbdcf328f83f071270609e824d6c5a"
  ostf_sha: "b978badb43243d3f1b85dde15b8e301a21f12254"
  fuelmain_sha: "f90d7d40ceff39ee6d7587b183c63bd00ed0743f"

Settings:
1 controller
1 compute
1 storage with CephOSD.

Ceph RDB for Cinder and Glance. QEMU, Neutron VLAN, no additional components installed.

Tags: ceph cinder
Kyrylo Romanenko (kromanenko)
Changed in mos:
milestone: none → 6.1
tags: added: ceph cinder
Changed in mos:
importance: Undecided → Critical
Ivan Kolodyazhny (e0ne)
Changed in mos:
assignee: nobody → Ivan Kolodyazhny (e0ne)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

cinder-backup can't initialize RBD driver and use default (Swift backend).

root@node-1:~# cinder backup-list | grep avail
| fdedf323-e37a-4238-a900-077a07266872 | e614bc39-0f2a-45d8-a68d-a3afa81e71f8 | available | None | 1 | 22 | volumebackups |

root@node-1:~# swift list volumebackups
volume_e614bc39-0f2a-45d8-a68d-a3afa81e71f8/20150407113556/az_nova_backup_fdedf323-e37a-4238-a900-077a07266872-00001
volume_e614bc39-0f2a-45d8-a68d-a3afa81e71f8/20150407113556/az_nova_backup_fdedf323-e37a-4238-a900-077a07266872-00002
...

Changed in mos:
status: New → Confirmed
assignee: Ivan Kolodyazhny (e0ne) → Fuel Library Team (fuel-library)
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

RBD driver should be configured for cinder-backup service

Vladimir Kuklin (vkuklin)
Changed in mos:
importance: Critical → High
Vladimir Kuklin (vkuklin)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 6.1
no longer affects: mos
Vladimir Kuklin (vkuklin)
summary: - Cinder backup service can not create volume backup
+ Cinder backup service can not create volume backup with ceph enabled
Vladimir Kuklin (vkuklin)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislav Makar (smakar)
Revision history for this message
Stanislav Makar (smakar) wrote :

I cant deploy such env config:

Settings:
1 controller
1 compute
1 storage with CephOSD.

Number of OSD nodes (1) cannot be less than the Ceph object replication factor (2). Please either assign ceph-osd role to more nodes, or reduce Ceph replication factor in the Settings tab.

I have checked on another env with ceph - all is working

We need more details about this error, what OS, what repl factor ?

if you could provide diagnostic snapshot it would be fine

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

I caught that bug with one OSD node and ReplicationFactor=1 respectively.
OS: Ubuntu 14.04.1.
Also i will try to reproduce to get diagnostic snapshot.

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :
Download full text (4.8 KiB)

Now i reproduced this bug on CentOS 6.5 environment also.

Settings:
1 controller + CephOSD.
1 compute + CephOSD.
1 storage CephOSD.

Ceph RDB for Cinder and Glance. QEMU, Neutron VLAN, no additional components installed.

Steps:

# service openstack-cinder-backup status
openstack-cinder-backup (pid 1384) is running...

# cinder service-list
+------------------+-------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
 +------------------+-------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | rbd:volumes | nova | enabled | up | 2015-04-15T12:18:43.000000 | None |
| cinder-scheduler | rbd:volumes | nova | enabled | up | 2015-04-15T12:18:44.000000 | None |
| cinder-volume | rbd:volumes | nova | enabled | up | 2015-04-15T12:18:49.000000 | None |
 +------------------+-------------+------+---------+-------+----------------------------+-----------------+

# cinder create 1 --display-name=VOLUME
+---------------------+--------------------------------------+
| Property | Value |
 +---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2015-04-15T12:07:41.100036 |
| display_description | None |
| display_name | VOLUME |
| encrypted | False |
| id | 2f28f60e-ed7a-456d-80e8-48d61d2eeaff |
| metadata | {} |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
 +---------------------+--------------------------------------+

# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 2f28f60e-ed7a-456d-80e8-48d61d2eeaff | available | VOLUME | 1 | None | false | |
 +--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

# cinder backup-create VOLUME --display-name=BKP
+-----------+--------------------------------------+
| Property | Value |
 +-----------+--------------------------------------+
| id | a6fe6dfe-8e7e-49bc-9e87-5da2ec10f5df |
| name | BKP |
| volume_id | 2f28f60e-ed7a-...

Read more...

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Cinder logs from Controller attached.

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

And diagnostic snapshot.

Revision history for this message
Stanislav Makar (smakar) wrote :

got such error but with two osd nodes
looking into it

Revision history for this message
Stanislav Makar (smakar) wrote :

root cause that cinder-backup was started when ceph cluster was not ready
Restarted it manually, working

<155>Apr 15 11:49:14 node-4 cinder-backup error connecting to ceph cluster
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd Traceback (most recent call last):
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 274, in check_for_setup_error
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd with RADOSClient(self):
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 239, in __init__
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd self.cluster, self.ioctx = driver._connect_to_rados(pool)
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/rbd.py", line 294, in _connect_to_rados
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd conffile=self.configuration.rbd_ceph_conf)
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/rados.py", line 242, in __init__
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd self.conf_read_file(conffile)
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd File "/usr/lib/python2.7/dist-packages/rados.py", line 291, in conf_read_file
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd raise make_ex(ret, "error calling conf_read_file")
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd Error: error calling conf_read_file: errno EINVAL
2015-04-15 11:49:14.256 29488 TRACE cinder.volume.drivers.rbd
<155>Apr 15 11:49:14 node-4 cinder-backup Error encountered during initialization of driver: RBDDriver.
<155>Apr 15 11:49:14 node-4 cinder-backup Bad or unexpected response from the storage volume backend API: error connecting to ceph cluster

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/173845

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/173845
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=54207adebca78b2e4b4f6bd967a766bd4a3cdc06
Submitter: Jenkins
Branch: master

commit 54207adebca78b2e4b4f6bd967a766bd4a3cdc06
Author: Stanislav Makar <email address hidden>
Date: Wed Apr 15 13:18:34 2015 +0000

    Restart cinder-backup service

    Restart cinder-backup service when Ceph mon is deployed. It is
    necessary due to cinder-backup can not work correctly even when
    ceph cluster is deployed later.

    Change-Id: I1aa7f23820c1866270a11bb4a2ae276869ef1af5
    Closes-bug: #1441109

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

There is new bug possible related to current one.
Cinder backup service: Volume backup of 50GB failed
https://bugs.launchpad.net/fuel/+bug/1446593

Vladimir Kuklin (vkuklin)
Changed in fuel:
status: Fix Committed → Triaged
Bartłomiej Piotrowski (bpiotrowski)
Changed in fuel:
assignee: Stanislav Makar (smakar) → Bartlomiej Piotrowski (bpiotrowski)
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Currently cinder backup service tested on
MOS VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2-6.1"
  api: "1.0"
  build_number: "329"
  build_id: "2015-04-20_22-54-31"
  nailgun_sha: "69fca3bada96a48ab19b969621e5128390b7052e"
  python-fuelclient_sha: "b4f1ddffd5263489090b65e662173e9e11aafd94"
  astute_sha: "bf1751a4fe0d912325e3b4af629126a59c1b2b51"
  fuellib_sha: "b878f1ae8e1d4c6f2f463e810b1f8014df5b22a5"
  ostf_sha: "df8db1f48f03b18126ce5ec65317a1eb83a5a95f"
  fuelmain_sha: "0c0e299dcc0276bb8262256148c6ee975857ee86"

Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Following functionality was checked.
     Centos Ubuntu
1 service starts pass pass
2 Backup-create Pass pass (~1.5 min per 1 GB)
3 backup-delete pass pass
4 Backup-list pass pass
5 backup-restore pass pass
6 backup-show pass pass

Bartłomiej Piotrowski (bpiotrowski)
Changed in fuel:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.