Error creating Volume from Image - Fuel 3.2 with Ceph

[Arminder Singh Girgla]

I'm running latest Fuel Web 3.2 and using Ceph RBD for volumes & images and Ceph Radosgw for objects storage.

i'm facing a peculiar problem trying to create volume from image. It doesn't seem to work. Although the volume is created instantly and I don't get any error, but it doesn't seem to fetch the image from glance since I see nothing in Ceph for image fetch. Also nothing in Ceph log for anything going into the volume.

When I try to boot from the resulting volume, it fails to boot and shows up as blank volume.

======================================
# cinder create --image-id 20e93c6c-bb04-469a-8e70-38db18feb526 --display-name "windows-test" --display-description "Volume for windows-test" --volume-type 86ed8306-17a8-4524-8d73-d39e9347b490 50

<0>Oct 25 07:18:29 node-6 ¿<156>cinder-cinder.api.v1.volumes AUDIT: Create volume of 50 GB
<0>Oct 25 07:18:29 node-6 ¿<156>cinder-cinder.api.v1.volumes AUDIT: vol={'volume_metadata': [], 'availability_zone': 'nova', 'terminated_at': None, 'updated_at': None, 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': '59980925-733e-4461-a495-59544162d75d', 'size': 50, 'user_id': u'fd581b3b1519472aaa7082fa6c3854a4', 'attach_time': None, 'display_description': u'Volume for windows-test', 'project_id': u'29e041604e124fcda7ca2fdc3a89e706', 'launched_at': None, 'scheduled_at': None, 'status': 'creating', 'volume_type_id': u'86ed8306-17a8-4524-8d73-d39e9347b490', 'deleted': False, 'provider_location': None, 'host': None, 'source_volid': None, 'provider_auth': None, 'display_name': u'windows-test', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 10, 25, 7, 18, 29, 931267), 'attach_status': 'detached', 'volume_type': <cinder.db.sqlalchemy.models.VolumeTypes object at 0x3c92d10>, 'metadata': {}}
<0>Oct 25 07:18:29 node-6 ¿<158>cinder-cinder.api.openstack.wsgi INFO: http://192.168.235.3:8776/v1/29e041604e124fcda7ca2fdc3a89e706/volumes returned with HTTP 200
<158>Oct 25 07:18:29 node-6 cinder-eventlet.wsgi.server INFO: 10.10.10.11 - - [25/Oct/2013 07:18:29] "POST /v1/29e041604e124fcda7ca2fdc3a89e706/volumes HTTP/1.1" 200 606 0.383747
<0>Oct 25 07:18:30 node-6 ¿<158>cinder-cinder.volume.manager INFO: volume volume-59980925-733e-4461-a495-59544162d75d: creating from image
<0>Oct 25 07:18:30 node-6 ¿<158>cinder-cinder.volume.manager INFO: volume volume-59980925-733e-4461-a495-59544162d75d: created successfully
======================================

- Arminder Singh Girgla

[Arminder Singh Girgla (arminder)]

Also to add that I'm running KVM as hypervisor.

[Andrew Woodward (xarses)]

Please perform these tests and paste output

https://github.com/Mirantis/fuel/tree/master/deployment/puppet/ceph#verifying-the-deployment

[Ryan Moe (rmoe)]

I was able to reproduce this issue on 3.2 CentOS + Ceph.

[Dmitry Borodaenko (angdraug)]

A workaround for this bug is to change glance_api_version in cinder.conf from 2 to 1. This will disable the copy on write behaviour that triggers the problem.

[Ryan Moe (rmoe)]

Ceph only supports CoW cloning of raw images. All images that are going to be used to create volumes have to be raw.

[Dmitry Borodaenko (angdraug)]

The problem is that the RBD driver in Cinder only checks that the image is accessible and doesn't check its attributes to confirm that it's in raw format.

Grizzly:
https://github.com/openstack/cinder/blob/stable/grizzly/cinder/volume/drivers/rbd.py#L243

Havana/Icehouse:
https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L716

[Arminder Singh Girgla (arminder)]

Thanks guys for your responses.

If you see the copy_image_to_volume function, its calling image_utils.fetch_to_raw function to fetch the image in raw format.

Shouldn't that take care of conversion in case the image is not in raw format? It doesn't seem to be calling qemu-img for conversion.

Grizzly:
https://github.com/openstack/cinder/blob/stable/grizzly/cinder/volume/drivers/rbd.py#L274

Havana/Icehouse:
https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L745

[Dmitry Borodaenko (angdraug)]

Yes, it would take care of conversion, but only if the image is copied, and not cloned, which is exactly what happens if you change glance API to v1 in cinder.conf. This code path is never reached if the rbd driver decides that the image is suitable for cloning, and it doesn't take image format into account when making that decision. I've raised this on the ceph-users mailing list, but haven't got any feedback from Ceph developers yet:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-November/005860.html

[Dmitry Borodaenko (angdraug)]

Patch for Cinder RBD backend:
https://review.openstack.org/58870

[Dmitry Borodaenko (angdraug)]

Related Cinder bug:
https://bugs.launchpad.net/cinder/+bug/1247998

[Dmitry Borodaenko (angdraug)]

The fix is merged to Cinder and will be included with Fuel 4.0.