Integrity check for images with checksum = "None"

[Ruediger Rissmann]

I am running juno with rbd storage.

When I try to create a volume from an image uploaded with

glance image-create --name imagename --location=http://some.location/image.raw --disk-format raw --container-format bare

the copy of the image fails with mageCopyFailure: Failed to copy image to volume: [Errno 32] Corrupt image download. Checksum was xxxxxxx expected None

The reason seems to be in
glanceclient/common/utils.py
def integrity_iter(iter, checksum):

Here the cinder-volume.log

ERROR oslo.messaging.rpc.dispatcher [req-67687cbf-42b8-478b-a326-ddd84a5d9d1b e2f75ca5e84744d4b22f71704809c240 3892d82ec1d34984ba92aa2f641aea89 - - -] Exception during message handling: Failed to copy image to volume: [Errno 32] Corrupt image download. Checksum was c2cf91777640c7b1d73a1003b73c4f39 expected None
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 381, in create_volume
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher _run_flow()
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 374, in _run_flow
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher flow_engine.run()
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 89, in run
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher for _state in self.run_iter():
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 137, in run_iter
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher misc.Failure.reraise_if_any(failures.values())
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 797, in reraise_if_any
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher failures[0].reraise()
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 804, in reraise
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher six.reraise(*self._exc_info)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 34, in _execute_task
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher result = task.execute(**arguments)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 638, in execute
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher **volume_spec)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 590, in _create_from_image
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher image_id, image_location, image_service)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 504, in _copy_image_to_volume
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex)
2015-03-18 13:45:32.729 2433 TRACE oslo.messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: [Errno 32] Corrupt image download. Checksum was c2cf91777640c7b1d73a1003b73c4f39 expected None

[Ruediger Rissmann (ruediger.rissmann)]

Here my fix:

https://review.openstack.org/#/c/165770

[Stuart McLaren (stuart-mclaren)]

I've tried to reproduce using just glance:

 1) create image
 $ glance image-create --name imagename --location=http://some.web.site --disk-format raw --container-format bare
 +------------------+--------------------------------------+
 | Property | Value |
 +------------------+--------------------------------------+
 | checksum | None |
 | container_format | bare |
 | created_at | 2015-04-09T14:55:20.000000 |
 | deleted | False |
 | deleted_at | None |
 | disk_format | raw |
 | id | a9384c14-6e4d-4dc2-bcff-c6d42a5b201b |
 | is_public | False |
 | min_disk | 0 |
 | min_ram | 0 |
 | name | imagename |
 | owner | fc3fe1a4154e401897495140b0eede33 |
 | protected | False |
 | size | 151 |
 | status | active |
 | updated_at | 2015-04-09T14:55:20.000000 |
 | virtual_size | None |
+------------------+--------------------------------------+

 2) show the image
 $ glance image-show a9384c14-6e4d-4dc2-bcff-c6d42a5b201b
 +------------------+--------------------------------------+
 | Property | Value |
 +------------------+--------------------------------------+
 | container_format | bare |
 | created_at | 2015-04-09T14:55:20.000000 |
 | deleted | False |
 | disk_format | raw |
 | id | a9384c14-6e4d-4dc2-bcff-c6d42a5b201b |
 | is_public | False |
 | min_disk | 0 |
 | min_ram | 0 |
 | name | imagename |
 | owner | fc3fe1a4154e401897495140b0eede33 |
 | protected | False |
 | size | 151 |
 | status | active |
 | updated_at | 2015-04-09T14:55:20.000000 |
 +------------------+--------------------------------------+

 3) download the image
 $ glance image-download a9384c14-6e4d-4dc2-bcff-c6d42a5b201b
 <html>
 some.web.page
</html>

This all seems to work fine.

What I suspect is happening is that cinder is 'reading' the checksum value of 'None' and passing that as a string into the glance client.

[Stuart McLaren (stuart-mclaren)]

On devstack I created a volume from the image:

 $ cinder create --image-id a9384c14-6e4d-4dc2-bcff-c6d42a5b201b --display-name vol1 1
 +---------------------+--------------------------------------+
 | Property | Value |
 +---------------------+--------------------------------------+
 | attachments | [] |
 | availability_zone | nova |
 | bootable | false |
 | created_at | 2015-04-10T09:27:41.148938 |
 | display_description | None |
 | display_name | vol1 |
 | encrypted | False |
 | id | 8d167acd-2220-45df-b256-8e66cee8b708 |
 | image_id | a9384c14-6e4d-4dc2-bcff-c6d42a5b201b |
 | metadata | {} |
 | multiattach | false |
 | size | 1 |
 | snapshot_id | None |
 | source_volid | None |
 | status | creating |
 | volume_type | lvmdriver-1 |
 +---------------------+--------------------------------------+

And that seemed to work fine.

I ran dd to verify that the image data was present on the volume (and it was):

 # dd if=/dev/mapper/stack--volumes--lvmdriver--1-volume--8d167acd--2220--45df--b256--8e66cee8b708 > /tmp/dd.out

Can you let me know the cinder command you used and the version of glance/cinder you are using?
Perhaps this is working ok with the latest code base?

[Ruediger Rissmann (ruediger.rissmann)]

I think the root cause for this bug is somewhere in cinder. The checksum should never be the string "None", right? Either it is a checksum value or None (undefined).
However it might be a good idea to have an additional check here as well.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Glance Bot (<email address hidden>) on branch: master
Review: https://review.openstack.org/165770

[Ian Cordasco (icordasc)]

This has been marked incomplete for nearly two years. Marking it as invalid until it can be reproduced against modern and supported versions of glanceclient/glance.