Create a volume from image fails

[Matthias Runge]

When trying to create a volume from an image, the creation is started, but stays in state "downloading"

[Matthias Runge (mrunge)]

[root@turing ~(keystone_admin)]# glance image-list+--------------------------------------+------+-------------+------------------+-----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+------+-------------+------------------+-----------+--------+
| 606b3e0e-ef37-41b7-9631-7b0bac5050a6 | f17 | qcow2 | bare | 251985920 | active |

cinder create --image-id 606b3e0e-ef37-41b7-9631-7b0bac5050a6 1

[root@turing ~(keystone_admin)]# cinder list
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
| cd93ff3b-e33f-434d-925f-efbc86ac7cd4 | downloading | None | 1 | None | False | |
+--------------------------------------+-------------+--------------+------+----

from cinder logs
2013-09-27 11:43:47.284 4473 ERROR cinder.volume.flows.create_volume [req-53582fed-05cc-4c76-b415-d9fc6c8c1541 ebf3ec8198f54ff0911d74ee76603723 2933aed883d9454ea34b0f6ebaf8547b] Failed to copy image 606b3e0e-ef37-41b7-9631-7b0bac5050a6 to volume: cd93ff3b-e33f-434d-925f-efbc86ac7cd4, error: qemu-img: /dev/mapper/cinder--volumes-volume--cd93ff3b--e33f--434d--925f--efbc86ac7cd4: error while converting raw: No space left on device

2013-09-27 11:43:47.285 4473 WARNING cinder.taskflow.utils [-] Activating 5 rollbacks due to <cinder.taskflow.utils.FlowFailure object at 0x4a07e50>.
2013-09-27 11:43:47.285 4473 ERROR cinder.taskflow.utils [-] Failed rolling back 4: cinder.volume.flows.create_volume.OnFailureRescheduleTask;volume:create==1.0 due to inner exception.
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils Traceback (most recent call last):
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils File "/usr/lib/python2.7/site-packages/cinder/taskflow/utils.py", line 353, in rollback
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils f(cause)
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils File "/usr/lib/python2.7/site-packages/cinder/taskflow/utils.py", line 186, in __call__
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils self.task.revert(self.context, self.result, cause)
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils File "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__init__.py", line 922, in revert
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils if not self._is_reschedulable(cause):
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils File "/usr/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__init__.py", line 832, in _is_reschedulable
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils exc_type, value = cause.exc_info()[:2]
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils TypeError: 'tuple' object is not callable
2013-09-27 11:43:47.285 4473 TRACE cinder.taskflow.utils

I assume, this error message is...

[Matthias Runge (mrunge)]

[root@turing cinder(keystone_admin)]# df -h /dev/mapper/cinder--volumes-volume--cd93ff3b--e33f--434d--925f--efbc86ac7cd4
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 0 3.9G 0% /dev

[Matthias Runge (mrunge)]

and the image taken to create was 250 mb.

[Matthias Runge (mrunge)]

an addition: creating a volume from iso works, from qcow2 does not work.

[Avishay Traeger (avishay-il)]

So one thing is there may be a bug in _is_reschedulable causing the TypeError in the Trace, but that's not the root cause here obviously.

When creating a volume from image where Cinder needs to convert to raw, it will first download the image to the directory specified in your cinder.conf file ('image_conversion_dir'), and then write to the volume. Do you have enough space in that directory?

[Matthias Runge (mrunge)]

Avishay, yes.
image_conversion_dir=/var/tmp

[root@turing ~]# df -h /var/tmp/
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 447G 212G 213G 50% /

[Matthias Runge (mrunge)]

and for reference, the image I took was this one:
http://berrange.fedorapeople.org/images/2012-11-15/f17-x86_64-openstack-sda.qcow2

[Avishay Traeger (avishay-il)]

So I downloaded the image and ran 'qemu-img info' on it:

image: f17-x86_64-openstack-sda.qcow2
file format: qcow2
virtual size: 9.8G (10486808576 bytes)
disk size: 240M
cluster_size: 65536

The virtual size is 9.8G, which means that you'll need a 10GB volume to hold it. You have a 1GB volume, so that explains the no space error.

Additionally, the trace shows an error in the _is_reschedulable function, which I see has been fixed.

So the main issue here in my opinion is that the state of the volume is 'downloading' instead of 'error'. Do you agree?

[Gabriel Hurley (gabriel-hurley)]

Looks like this is a Cinder issue, not something that can be reasonably fixed in Horizon.

[Matthias Runge (mrunge)]

Well for cinder: I was more concerned about a general issue here (not being able to create a volume from a qcow2 (or else) at all).

Regarding the message: error would be better, including "no space left on device".

I was plainly unaware, that this image takes nearly 10 gigs. -> user error.

[John Griffith (john-griffith)]

Def should improve error messaging here, I would say user error but it brings up a deficiency in our error reporting I think. It would be nice to clean this up for I-1.

[Ganpat Agarwal (gans-developer)]

i am trying to create a volume from image via iSCSI connection.

Volume size is 10 GB.

image : http://uec-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

qemu-img info 6b259bc5-2250-4f7a-b63e-104598ab4066
image: 6b259bc5-2250-4f7a-b63e-104598ab4066
file format: qcow2
virtual size: 2.0G (2147483648 bytes)
disk size: 243M
cluster_size: 65536

The operation is failing at /opt/stack/cinder/cinder/image/image_utils.py(200)convert_image() :

Command : /etc/cinder/rootwrap.conf qemu-img convert -O raw /tmp/tmpCBuvvX /dev/disk/by-path/ip-10.3.3.102:3260-iscsi-iqn.1992-04.com.emc:cx.apm00123901876.a8-lun-0

Error : qemu-img: /dev/disk/by-path/ip-10.3.3.102:3260-iscsi-iqn.1992-04.com.emc:cx.apm00123901876.a8-lun-0: error while converting raw: No space left on device

i did a ls on the destination path :

root@node2:~# ls -l /dev/disk/by-path/ip-10.3.3.102:3260-iscsi-iqn.1992-04.com.emc:cx.apm00123901876.a8-lun-0

lrwxrwxrwx 1 root root 9 Nov 10 19:34 /dev/disk/by-path/ip-10.3.3.102:3260-iscsi-iqn.1992-04.com.emc:cx.apm00123901876.a8-lun-0 -> ../../sdd

Someone please try to help me understand the root cause of this error.

[Ganpat Agarwal (gans-developer)]

I tried to perform copy_image_to_volume by using the default "LVMVolumeDriver" and was a success.

I re-phrase my question now : "Has anyone got success in copy_image_to_volume via an iSCSI driver?"

[Xing Yang (xing-yang)]

I can do copy_image_to_volume via my iSCSI driver. The error says "No space left on device". Can you check the disk space on your host?

[Eric Harney (eharney)]

Do the changes in https://bugs.launchpad.net/nova/+bug/1235358 help here any?