Incorrect ImageTooBig exception, with Ceph COW, cinder checks for free disk space even if no conversion is needed

[zhang wenjian]

In Ocata volume/flows/manager/create_volume.py, I have set glance and cinder backend to a same ceph pool.

class CreateVolumeFromSpecTask(flow_utils.CinderTask):
    ....

    def _create_from_image(self, context, volume,
                           image_location, image_id, image_meta,
                           image_service, **kwargs):
    ....

        # NOTE(e0ne): check for free space in image_conversion_dir before
        # image downloading.
        if (CONF.image_conversion_dir and not
                os.path.exists(CONF.image_conversion_dir)):
            os.makedirs(CONF.image_conversion_dir)
        image_utils.check_available_space(CONF.image_conversion_dir,
                                          image_meta['size'], image_id)

Is check_available_space need to called first here? When I set glance & cinder backend to same ceph pool, _create_from_image will call self.driver.clone_image directly without image_conversion_dir.

[Cheng Li (shcli)]

We may be able to move the check down, right before this line https://github.com/openstack/cinder/blob/03c85d96511c9c54630019baad115aa2cba746ba/cinder/volume/flows/manager/create_volume.py#L841
Please feel free to try it out

[Alexander Bozhenko (alexbozhenko)]

@e0ne, Ivan, could you please take a look at this?|
We do not need to check if image fits /var/lib/cinder/convertion if this is a volume-backed image, or it is a image cached in glance, right?

[Teun Docter (teun-docter)]

We ran into this as well on our Pike test setups. We use VM's for testing and they have small disks because we use Ceph as back ends for Cinder and Glance.

The following stack trace can be observed in the cinder volume log:

[ 2722] WARNING: Flow 'volume_create_manager' (db419b72-f0ef-4089-9e41-1d683bcd72bc) transitioned into state 'REVERTED' from state 'RUNNING'
[ 2722] WARNING: Function 'cinder.service.Service.report_state' run outlasted interval by 32.35 sec
[ 2722] ERROR: Exception during message handling: ImageTooBig: Image 99aaada6-c8d5-411f-b0ae-501d5fd7c023 size exceeded available disk space: There is no sp
ace to convert image. Requested: 13697548288, available: 4787212288
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
>> res = self.dispatcher.dispatch(message)
>> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
>> return self._do_dispatch(endpoint, method, ctxt, args)
>> File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
>> result = func(ctxt, **new_args)
>> File "<string>", line 2, in create_volume
>> File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
>> result = f(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 651, in create_volume
>> _run_flow()
>> File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 643, in _run_flow
>> flow_engine.run()
>> File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
>> for _state in self.run_iter(timeout=timeout):
>> File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
>> failure.Failure.reraise_if_any(er_failures)
>> File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
>> failures[0].reraise()
>> File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
>> six.reraise(*self._exc_info)
>> File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
>> result = task.execute(**arguments)
>> File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 951, in execute
>> **volume_spec)
>> File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 856, in _create_from_image
>> exception=err)
>> File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>> self.force_reraise()
>> File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>> six.reraise(self.type_, self.value, self.tb)
>> File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 848, in _create_from_image
>> image_meta['size'...

[Tadas Ustinavičius (tadas-u)]

I've met this problem too.
I've found why this happens and how to fix it.

_create_from_image() function checks for free space in conversion path (by default /var/lib/cinder/conversion) even if you are building volume from volume snapshot (no download and conversion by cinder required).

If your cinder server has less free space than the image you specified to create volume from, exception will be called by check_available_space() in image_utils module.

This patch:
http://paste.ubuntu.com/26079789/
solves the problem for cinder if you are using "allowed_direct_url_schemes = cinder" in your cinder.conf (using image snapshots, as volumes).

[Tadas Ustinavičius (tadas-u)]

Forgot to mention, that the patch is for

cinder/volume/flows/manager/create_volume.py

file

[Crazik (crazik)]

You can try some workaround in OS without changing code.
Just create big sparse file and mount on conversion path.

[Tadas Ustinavičius (tadas-u)]

Mounting sparse file won't do good - conversion is still needed in some occasions (for example images are stored in qcow2 format rather than raw).

[Crazik (crazik)]

It's true, but in my case I use only raw images.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/524331
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=30bb0dfbe77efcfcb2330f1758d229310e52ff74
Submitter: Zuul
Branch: master

commit 30bb0dfbe77efcfcb2330f1758d229310e52ff74
Author: Tadas Ustinavičius <email address hidden>
Date: Thu Nov 30 22:06:46 2017 +0200

    Fix for volume from image snapshot free space issue

    create_from_image() function checks for free space in conversion path
    (by default /var/lib/cinder/conversion) even if cinder is creating
    volume from image snapshot.

    If /var/lib/cinder/conversion has less free space than the size of the
    image volume is created from, exception will be thrown by
    check_available_space() function in image_utils module.

    This patch checks if cinder is configured with
    allowed_direct_url_schemes = cinder
    If so, cinder should not check free space in conversion path because
    volume is created from image snapshot.

    Closes-Bug: #1683228
    Change-Id: I6330ed31dc9336c9d3d09c2d43c8f4913744e9a2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/526622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526622
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=4320c7874cff3b56c1af6dcd45a11e15195237ba
Submitter: Zuul
Branch: stable/pike

commit 4320c7874cff3b56c1af6dcd45a11e15195237ba
Author: Tadas Ustinavičius <email address hidden>
Date: Thu Nov 30 22:06:46 2017 +0200

    Fix for volume from image snapshot free space issue

    create_from_image() function checks for free space in conversion path
    (by default /var/lib/cinder/conversion) even if cinder is creating
    volume from image snapshot.

    If /var/lib/cinder/conversion has less free space than the size of the
    image volume is created from, exception will be thrown by
    check_available_space() function in image_utils module.

    This patch checks if cinder is configured with
    allowed_direct_url_schemes = cinder
    If so, cinder should not check free space in conversion path because
    volume is created from image snapshot.

    Closes-Bug: #1683228
    Change-Id: I6330ed31dc9336c9d3d09c2d43c8f4913744e9a2
    (cherry picked from commit 30bb0dfbe77efcfcb2330f1758d229310e52ff74)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.0.2 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/571708

[Carlos Konstanski (ckonstanski)]

This is also a problem in queens, and it's bigger than ceph. A link to a relevant thread on ask.openstack.org:

https://ask.openstack.org/en/question/112812/cant-convert-image-to-volume/?answer=114733#post-id-114733

I have found that the destination where cinder wants to copy the image for conversion is /dev/mapper. Is this not bizarre? That's inside a pseudofilesystem.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571708
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=1c899cf56cfb9ccf25ad3849a3eb1afbbb1867fe
Submitter: Zuul
Branch: stable/ocata

commit 1c899cf56cfb9ccf25ad3849a3eb1afbbb1867fe
Author: Tadas Ustinavičius <email address hidden>
Date: Thu Nov 30 22:06:46 2017 +0200

    Fix for volume from image snapshot free space issue

    create_from_image() function checks for free space in conversion path
    (by default /var/lib/cinder/conversion) even if cinder is creating
    volume from image snapshot.

    If /var/lib/cinder/conversion has less free space than the size of the
    image volume is created from, exception will be thrown by
    check_available_space() function in image_utils module.

    This patch checks if cinder is configured with
    allowed_direct_url_schemes = cinder
    If so, cinder should not check free space in conversion path because
    volume is created from image snapshot.

    Closes-Bug: #1683228
    Change-Id: I6330ed31dc9336c9d3d09c2d43c8f4913744e9a2
    (cherry picked from commit 30bb0dfbe77efcfcb2330f1758d229310e52ff74)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 10.0.8 release.