Glance does not return image MD5

[Vinicius G Ferreira]

I'm trying to download an OpenStack image from glance using only the Openstack Python SDK, but I only get this error:

Traceback (most recent call last):
  File "/home/openstack/discovery/discovery.py", line 222, in <module>
    main(sys.argv[1:])
  File "/home/openstack/discovery/discovery.py", line 117, in main
    image_service.download_image(image)
  File "/usr/local/lib/python2.7/dist-packages/openstack/image/v2/_proxy.py", line 72, in download_image
    return image.download(self.session)
  File "/usr/local/lib/python2.7/dist-packages/openstack/image/v2/image.py", line 166, in download
    checksum = resp.headers["Content-MD5"]
  File "/usr/local/lib/python2.7/dist-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-md5'

The weird part is that if I run the code using an IDE (PyCharm with remote debug) or as a script (python script.py -i ...) I get the error, but if I run each line using a python interpreter (ipython/python) the error does not happen! Have no idea why.

Here is the code I'm using:

...
image_name = node.name + "_" + time.strftime("%Y-%m-%d_%H-%M-%S")
print "Getting data from", node.name
compute_service.create_server_image(node, image_name)
image = image_service.find_image(image_name)
image_service.wait_for_status(image, 'active')
fileName = "%s.img" % image.name

with open(str(fileName), 'w+') as imgFile:
    imgFile.write(image.download(conn.image.session))
...

This code ends up calling the API in this file /usr/local/lib/python2.7/dist-packages/openstack/image/v2/image.py, with this method:

def download(self, session):
    """Download the data contained in an image"""
    # TODO(briancurtin): This method should probably offload the get
    # operation into another thread or something of that nature.
    url = utils.urljoin(self.base_path, self.id, 'file')
    resp = session.get(url, endpoint_filter=self.service)

    checksum = resp.headers["Content-MD5"]
    digest = hashlib.md5(resp.content).hexdigest()
    if digest != checksum:
        raise exceptions.InvalidResponse("checksum mismatch")

    return resp.content

The resp.headers variable has no key "Content-MD5". This is the value I found for it:

{'Date': 'Thu, 01 Sep 2016 20:17:01 GMT', 'Transfer-Encoding': 'chunked',
 'Connection': 'keep-alive', 'Content-Type': 'application/octet-stream',
 'X-Openstack-Request-Id': 'req-9eb16897-1398-4ab2-9cd4-45706e92819c'}

But according to the REST API documentationm the response should return with the key Content-MD5: http://developer.openstack.org/api-ref/image/v2/?expanded=download-binary-image-data-detail

If I just comment the MD5 check the download works fine, but this is inside the SDK so I can't/shouldn't change it. Anyone have any suggestion on how to achieve this using the OpenStack Python SDK? Is this an SDK bug?

[Brian Curtin (brian.curtin)]

That is interesting that it would work in some ways but not in others. Can you put the following two lines at the top of your script, and then run it and send the sdk.log file to me at <email address hidden>? It seems pretty odd that the response data would be different depending on where it's run.

from openstack import utils
utils.enable_logging(debug=True, path="sdk.log")

[Ian Cordasco (icordasc)]

This definitely seems to be a server bug but I'm not sure why.

Could you also provide the headers from when you don't encounter the error? I suspect that might provide some insight.

[Vinicius G Ferreira (vini-g-fer)]

Problem now seems to be happening pretty much in all attempts to download the glance image. I was able to successfully download only once, and I received this header:

{'Content-Length': '11206656', 'Content-Md5': '415384c0cc42b1af13829b33cbcfa783', 'Connection': 'keep-alive', 'Date': 'Fri, 02 Sep 2016 18:15:10 GMT', 'Content-Type': 'application/octet-stream', 'X-Openstack-Request-Id': 'req-580c4a25-8579-474d-8af0-0b1384997d4a'}

I sent the log to Brian email, and also upload it to this dropbox link: https://www.dropbox.com/s/coyy2ijdawzbwdq/sdk.log?dl=0

[Vinicius G Ferreira (vini-g-fer)]

Any news on this? Was the log useful in any way?

[Brian Curtin (brian.curtin)]

Hey, sorry for not getting back to you quicker. The log shows that on that particular request, the server's response doesn't include the MD5, so I think Ian is right that this is more of a Glance problem than it is an SDK problem. I'm going to add Glance in here so perhaps they can take a look.

[Cyril Roelandt (cyril-roelandt)]

So apparently, the Content-MD5 field is not always set because of an issue in Pylons:

https://github.com/openstack/glance/blob/master/glance/api/v2/image_data.py#L317

So, if image.checksum is not set, the field will be missing. Could you please post a *complete* code sample (similar to what is in your original post) that I could just chmod +x and run?

[Vinicius G Ferreira (vini-g-fer)]

Sure, here is a short code sample that reproduces the problem for me:

from openstack import connection

#Adjust these credentials according to your openstack installation
username = "demo"
password = "openstack2016"
auth_url_sdk = "http://10.10.0.1:5000/v3"
user_domain_name = "Default"
project_domain_name = "Default"
project_name = "demo"

conn = connection.Connection(
        auth_url=auth_url_sdk,
        project_name=project_name,
        project_domain_name=project_domain_name,
        username=username,
        user_domain_name=user_domain_name,
        password=password)
image_service = conn.image
compute_service = conn.compute

#Using cirros image, which is usually pre-built with openstack/devstack
image = image_service.find_image('cirros-0.3.4-x86_64-uec')
flavor = compute_service.find_flavor('m1.tiny')

user_instance = compute_service.create_server(
name="TEST",
image_id=image.id,
flavor_id=flavor.id)
compute_service.wait_for_status(user_instance, status='ACTIVE')

compute_service.create_server_image(user_instance, "Snapshot")
image = image_service.find_image("Snapshot")
image_service.wait_for_status(image, 'active')

filename = "%s.img" % image.name
with open(str(filename), 'w+') as imgFile:
    for chunk in image.download(conn.image.session):
        imgFile.write(chunk)

[Cyril Roelandt (cyril-roelandt)]

OK, I can confirm this fails with a KeyError, as shown previously.

The Content-MD5 header is nowhere to be found, despite being set in download() in glance. One could change openstack/image/v2/image.py to work around the issue by doing:

- checksum = resp.headers["Content-MD5"]
+ checksum = self.checksum

in the download() function, but this is an ugly workaround I'm afraid.

I think it may not be an issue in Glance. Could you try to run a curl command that does the same thing as image.download(conn.image.session)? Something along the lines of:

export OS_TOKEN=...
export IMAGEID=...
curl -v -X GET \
        -H "X-Auth-Token: $OS_TOKEN" \
        http://10.0.2.15:9292/v2/images/$IMAGEID/file > /dev/null

I get "< Content-Md5: db0f920c8f85d215fe63a53693f31950" in the response.

[Brian Curtin (brian.curtin)]

The log file in message #3 above shows that the response to the GET he made does not contain the Content-MD5 header, but the response did return the image data (once you see the GET, the 40,000+ lines after that are the image itself--I'm working on a KSA fix so it won't log that). That was logged inside keystoneauth and before the data was touched by OpenStack SDK..

If it's documented to return the Content-MD5 header but we don't always get this header, we can do two things: fail the download with some exception because we can't trust that the content we got back is what was expected (this is bad), or just don't check for the Content-MD5 header at all (this is also bad).

[Vinicius G Ferreira (vini-g-fer)]

The

checksum = self.checksum

change suggested by Cyril is really such an ugly workaround? Another alternative I believe would be good: perform the MD5 check only if we actually receive the Content-MD5 header (without throwing any exception). I know it's not ideal. That's why I'm asking about the

checksum = self.checksum

solution.

[Brian Curtin (brian.curtin)]

That would just set it to None. We're not doing that.

[Brian Curtin (brian.curtin)]

I guess we need to check for the existence of Content-MD5 in a response and only use it when it exists. When it doesn't exist we should log a warning that it's not being validated, and then hopefully in the future we just always receive Content-MD5.

[wangxiyuan (wangxiyuan)]

Could you show your image first to see whether it contains the "checksum" property or not?

$glance image-show IMAGEID

in Glance, it will not return Content-MD5 in response headers if the image doesn't contain "checksum":
https://github.com/openstack/glance/blob/master/glance/api/v2/image_data.py#L317-L318

If so, you should find the reason why the image doesn't contain "checksum", and the SDK should catch this KeyError.

[wangxiyuan (wangxiyuan)]

and this bug maybe do some help:
https://github.com/Pylons/webob/issues/86

[Cyril Roelandt (cyril-roelandt)]

Reading the code, I really do not understand why Glance would not return the Content-MD5 header. Also, I'm unable to reproduce this issue with a curl command, even though it seems to happen when running the given Python sample.

What version of Glance are you using?

[Vinicius G Ferreira (vini-g-fer)]

Running "glance --version" returned me version 2.3.0. I'm using devstack with master branch from github, latest commit on Aug 11th (so Mitaka release).

Running glance image-show IMAGEID also showed that the image (created using create_server_image method) do have a checksum value.

[Brian Curtin (brian.curtin)]

I'm working on a change to SDK that will try to find the Content-MD5 header first, but if it doesn't find it it'll make an additional GET to retrieve the details and look for the `checksum` field in that response to use. One more test case and I'll have a review up.

[Vinicius G Ferreira (vini-g-fer)]

I don't know if this is helpful in anyway, but: this MD5 error doesn't happen when I use the python glance client. I am able to download the image normally. I think this doesn't matter as both openstacksdk and glance client use the same REST API in theory.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/367459

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/367459
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=759651f4a9eae2ba546f46613550a4cb10ddd964
Submitter: Jenkins
Branch: master

commit 759651f4a9eae2ba546f46613550a4cb10ddd964
Author: Brian Curtin <email address hidden>
Date: Thu Sep 8 11:18:18 2016 -0400

    Obtain Image checksum via additional GET

    It seems that some cases in the Image v2 GET /{image_id}/file call don't
    appear to be returning the Content-MD5 header at all times, yet our
    Image.download method is depending on that header in order to figure out
    if the image data is valid to return. Another way to get that value was
    mentioned as doing a GET on /{image_id} and looking at the `checksum`
    field, which we had already been receiving, but not using directly
    because of how the download method works.

    We now check for the header first, and if it's not there we make an
    additional call to get the image details and find the checksum that way.
    If *that* also does not return a checksum, we return the data anyway and
    log a warning message that we're not able to verify the integrity of the
    returned data.

    Change-Id: Ied68b7c6ca94155ed308911bedd34f0445c40050
    Related-Bug: 1619675