[OSSA 2015-004] Image data remains in backend after deleting the image created using task api (import-from) (CVE-2015-1881)

[Abhishek Kekane]

Trying to delete image created using task api (import-from) image gets deleted from the database, but image data remains in the backend.

Steps to reproduce:
1. Create image using task api

$ curl -i -X POST -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/json' -H 'Accept-Encoding: gzip, deflate, compress' -H 'Accept: */*' -H 'X-Auth-Token: 35a9e49237b74eddbe5057eb434b3f9e' -d '{"type": "import", "input": {"import_from": "http://releases.ubuntu.com/14.10/ubuntu-14.10-server-i386.iso", "import_from_format": "raw", "image_properties": {"disk_format": "raw", "container_format": "bare", "name": "task_image"}}}' http://10.69.4.176:9292/v2/tasks

2. wait until image becomes active.
3. Confirm image is in active state.
   $ glance image-list
4. Delete the image
   $ glance image-delete <image-id>
5. Verify image-list does not show deleted image
   $ glance image-list

Image gets deleted from the database but image data presents in the backend.

Problem:
Import task does not update the location of the image and it remains None even image becomes active.
Location entry is not added in the database in image_locations table.

While deleting the image it checks if location is present for image [1][2] then only it deletes that image data from that location.

[1] v1: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1066
[2] v2: https://github.com/openstack/glance/blob/master/glance/location.py#L361

This issue is reproducible in stable/juno as well as in current master.

Note: You need to replace auth_token in above curl command, otherwise it will raise error for authentication failure.
(Use 'keystone token-get' command to generate the new token)

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Thierry Carrez (ttx)]

So the attack scenario here would be to create/delete a lot of those and DoS the image backend by filling it up ? The half-deleted stuff not counting in any quota ?

[Nikhil Komawar (nikhil-komawar)]

Thierry: I think you put that concisely.

[Abhishek Kekane (abhishek-kekane)]

Hi Thierry,

You are right.

Attack scenario here is to create/delete a lot of those and DoS the image backend by filling it up.

[Abhishek Kekane (abhishek-kekane)]

In the attached patch, I am adding location information to the image.

[Flavio Percoco (flaper87)]

There's a bug there indeed. Thanks for the report.

@Abhishek, may I know why you deleted the code that checks whether the image is in `saving` state or not?

I agree that code is not good - there's a race condition there - but we shouldn't remove it in this patch.

[Abhishek Kekane (abhishek-kekane)]

Hi Flavio,

I have considered your comment and made changes accordingly.
Please have a look at new patch.

Thank you for review.

[Nikhil Komawar (nikhil-komawar)]

I find the second patch okay. If another core is okay with it, we can move ahead with a review for this.

Thanks for the effort, Abhishek!

[Tristan Cacqueray (tristan-cacqueray)]

Assuming the task api is always presents and this affects Icehouse, here is impact description draft:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Versions: up to 2014.1.3 and 2014.2 versions up to 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may leaks images data in the backend resulting in potential resources exhaustion and denial of service. All glance setups are affected.

[Flavio Percoco (flaper87)]

The second patch looks OK to me.

Abhishek, I'll apply it for you and move it forward.

Thanks.

[Abhishek Kekane (abhishek-kekane)]

Thank you Flavio and Nikhil for review,

Also I think Import Task was added in Juno, so this issue will not affect Icehouse.

[Abhishek Kekane (abhishek-kekane)]

Hi,

In stable/juno if image is deleted while uploading is in progress (image is in saving state) then image_repo.get() will raise NotFound exception which is not caught, so it will not delete image data from the backend.

https://review.openstack.org/#/c/156553/1/glance/common/scripts/image_import/main.py

The code should be like,

try:
        # NOTE: Check if the Image is not deleted after setting the data
        # before saving the active image. Here if image status is
        # saving, then new_image is saved as it contains updated location,
        # size, virtual_size and checksum information and the status of
        # new_image is already set to active in set_image_data() call.
        image = image_repo.get(image_id)
        if image.status == 'saving':
            image_repo.save(new_image)
            return image_id
        else:
            msg = _("The Image %(image_id)s object being created by this task "
                    "%(task_id)s, is no longer in valid status for further "
                    "processing.") % {"image_id": image_id,
                                      "task_id": task_id}
            raise exception.Conflict(msg)
except (exception.Conflict, exception.NotFound):
        with excutils.save_and_reraise_exception():
            if new_image.locations:
                for location in new_image.locations:
                    store_utils.delete_image_location_from_backend(
                        new_image.context,
                        image_id,
                        location)

Should I report this issue separately or it should be fixed in this same security bug.

[Flavio Percoco (flaper87)]

If the patch is changing the current "expected" behavior - as in, it's fixing the bug but "unfixing" other things- then it should be done in this same bug. Otherwise, it should happen in a separate issue.

[Abhishek Kekane (abhishek-kekane)]

Hi,

To fix this problem completely in stable juno we should also backport https://review.openstack.org/#/c/122427/ along with https://review.openstack.org/#/c/156553 patch.

[Tristan Cacqueray (tristan-cacqueray)]

Hum, so the cat is now out of the bag since the fix have been submitted to gerrit under the change-Id: Ie389de6538a9b98dc51c7d781b81b3ab10b83842

Though I guess this did not covered the delete during upload case mentioned in comment #13. I'm afraid we'll have to open this bug nonetheless since it is referenced in the public review, any chance to have a quick fix?

[Tristan Cacqueray (tristan-cacqueray)]

Oh and thanks for the impact description review, here is an update with proper affected versions:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Affects: 2014.2 versions through 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may leaks images data in the backend resulting in potential resources exhaustion and denial of service. All glance setups are affected.

[Jeremy Stanley (fungi)]

Grammatically it would be "...may leak image data..." but I think the sentence is misleading (this is not an information leak vulnerability, it's a resource consumption vulnerability). How about "...may accumulate untracked image data..." instead?

Also, "...potential resource exhaustion..." later in the same sentence.

Otherwise Tristan's updated impact description in comment #17 looks great. Thanks!

[Abhishek Kekane (abhishek-kekane)]

Hi Tristan,

For the issue I have mentioned in comment #13, I have reported new private bug https://bugs.launchpad.net/glance/+bug/1422716

To fix #1422716 issue, we simply need to cherrypick I4f7d1aa103f4ce7abf4026e7097b9e76c24135fa commit to stable/juno.

[Tristan Cacqueray (tristan-cacqueray)]

Many thanks Abhishek!

To sumup, the required change to fix *all* reported import task vulnerabilities are:

Kilo:
https://review.openstack.org/#/c/156493/
https://review.openstack.org/#/c/122427/

Juno:
https://review.openstack.org/#/c/156553/
[backports of 122427]

If yes, then we should open this bug and 1422716 then move on the CVE request/publish ossa.

@fungi, thanks for the review, it makes much sense, here is the updated impact description:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Affects: 2014.2 versions through 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may accumulate untracked image data in the backend resulting in potential resource exhaustion and denial of service. All glance setups are affected.

[Thierry Carrez (ttx)]

Impact desc +1

[Thierry Carrez (ttx)]

Public patches posted

[Stuart McLaren (stuart-mclaren)]

Minor:

> All glance setups are affected.

If this is specific to image import (import-from) then it only affects v2 (v1 has no import functionality).