[OSSA 2015-004] Image data remains in backend after deleting the image created using task api (import-from) (CVE-2015-1881)

Bug #1420696 reported by Abhishek Kekane on 2015-02-11
276
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Critical
Abhishek Kekane
Icehouse
Undecided
Unassigned
Juno
Critical
Abhishek Kekane
OpenStack Security Advisory
High
Tristan Cacqueray

Bug Description

Trying to delete image created using task api (import-from) image gets deleted from the database, but image data remains in the backend.

Steps to reproduce:
1. Create image using task api

$ curl -i -X POST -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/json' -H 'Accept-Encoding: gzip, deflate, compress' -H 'Accept: */*' -H 'X-Auth-Token: 35a9e49237b74eddbe5057eb434b3f9e' -d '{"type": "import", "input": {"import_from": "http://releases.ubuntu.com/14.10/ubuntu-14.10-server-i386.iso", "import_from_format": "raw", "image_properties": {"disk_format": "raw", "container_format": "bare", "name": "task_image"}}}' http://10.69.4.176:9292/v2/tasks

2. wait until image becomes active.
3. Confirm image is in active state.
   $ glance image-list
4. Delete the image
   $ glance image-delete <image-id>
5. Verify image-list does not show deleted image
   $ glance image-list

Image gets deleted from the database but image data presents in the backend.

Problem:
Import task does not update the location of the image and it remains None even image becomes active.
Location entry is not added in the database in image_locations table.

While deleting the image it checks if location is present for image [1][2] then only it deletes that image data from that location.

[1] v1: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1066
[2] v2: https://github.com/openstack/glance/blob/master/glance/location.py#L361

This issue is reproducible in stable/juno as well as in current master.

Note: You need to replace auth_token in above curl command, otherwise it will raise error for authentication failure.
(Use 'keystone token-get' command to generate the new token)

Tags: ntt Edit Tag help
Changed in glance:
assignee: nobody → Abhishek Kekane (abhishek-kekane)
description: updated

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
description: updated
Thierry Carrez (ttx) wrote :

So the attack scenario here would be to create/delete a lot of those and DoS the image backend by filling it up ? The half-deleted stuff not counting in any quota ?

Nikhil Komawar (nikhil-komawar) wrote :

Thierry: I think you put that concisely.

Changed in glance:
status: New → Triaged
importance: Undecided → Critical
milestone: none → kilo-3

Hi Thierry,

You are right.

Attack scenario here is to create/delete a lot of those and DoS the image backend by filling it up.

In the attached patch, I am adding location information to the image.

Flavio Percoco (flaper87) wrote :

There's a bug there indeed. Thanks for the report.

@Abhishek, may I know why you deleted the code that checks whether the image is in `saving` state or not?

I agree that code is not good - there's a race condition there - but we shouldn't remove it in this patch.

Hi Flavio,

I have considered your comment and made changes accordingly.
Please have a look at new patch.

Thank you for review.

tags: added: juno-backport-potential
Changed in ossa:
status: Incomplete → Confirmed
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
importance: Undecided → High
Nikhil Komawar (nikhil-komawar) wrote :

I find the second patch okay. If another core is okay with it, we can move ahead with a review for this.

Thanks for the effort, Abhishek!

Assuming the task api is always presents and this affects Icehouse, here is impact description draft:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Versions: up to 2014.1.3 and 2014.2 versions up to 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may leaks images data in the backend resulting in potential resources exhaustion and denial of service. All glance setups are affected.

Changed in ossa:
status: Confirmed → Triaged
Flavio Percoco (flaper87) wrote :

The second patch looks OK to me.

Abhishek, I'll apply it for you and move it forward.

Thanks.

Thank you Flavio and Nikhil for review,

Also I think Import Task was added in Juno, so this issue will not affect Icehouse.

Hi,

In stable/juno if image is deleted while uploading is in progress (image is in saving state) then image_repo.get() will raise NotFound exception which is not caught, so it will not delete image data from the backend.

https://review.openstack.org/#/c/156553/1/glance/common/scripts/image_import/main.py

The code should be like,

try:
        # NOTE: Check if the Image is not deleted after setting the data
        # before saving the active image. Here if image status is
        # saving, then new_image is saved as it contains updated location,
        # size, virtual_size and checksum information and the status of
        # new_image is already set to active in set_image_data() call.
        image = image_repo.get(image_id)
        if image.status == 'saving':
            image_repo.save(new_image)
            return image_id
        else:
            msg = _("The Image %(image_id)s object being created by this task "
                    "%(task_id)s, is no longer in valid status for further "
                    "processing.") % {"image_id": image_id,
                                      "task_id": task_id}
            raise exception.Conflict(msg)
except (exception.Conflict, exception.NotFound):
        with excutils.save_and_reraise_exception():
            if new_image.locations:
                for location in new_image.locations:
                    store_utils.delete_image_location_from_backend(
                        new_image.context,
                        image_id,
                        location)

Should I report this issue separately or it should be fixed in this same security bug.

Flavio Percoco (flaper87) wrote :

If the patch is changing the current "expected" behavior - as in, it's fixing the bug but "unfixing" other things- then it should be done in this same bug. Otherwise, it should happen in a separate issue.

Hi,

To fix this problem completely in stable juno we should also backport https://review.openstack.org/#/c/122427/ along with https://review.openstack.org/#/c/156553 patch.

Hum, so the cat is now out of the bag since the fix have been submitted to gerrit under the change-Id: Ie389de6538a9b98dc51c7d781b81b3ab10b83842

Though I guess this did not covered the delete during upload case mentioned in comment #13. I'm afraid we'll have to open this bug nonetheless since it is referenced in the public review, any chance to have a quick fix?

Oh and thanks for the impact description review, here is an update with proper affected versions:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Affects: 2014.2 versions through 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may leaks images data in the backend resulting in potential resources exhaustion and denial of service. All glance setups are affected.

Jeremy Stanley (fungi) wrote :

Grammatically it would be "...may leak image data..." but I think the sentence is misleading (this is not an information leak vulnerability, it's a resource consumption vulnerability). How about "...may accumulate untracked image data..." instead?

Also, "...potential resource exhaustion..." later in the same sentence.

Otherwise Tristan's updated impact description in comment #17 looks great. Thanks!

Hi Tristan,

For the issue I have mentioned in comment #13, I have reported new private bug https://bugs.launchpad.net/glance/+bug/1422716

To fix #1422716 issue, we simply need to cherrypick I4f7d1aa103f4ce7abf4026e7097b9e76c24135fa commit to stable/juno.

Many thanks Abhishek!

To sumup, the required change to fix *all* reported import task vulnerabilities are:

Kilo:
https://review.openstack.org/#/c/156493/
https://review.openstack.org/#/c/122427/

Juno:
https://review.openstack.org/#/c/156553/
[backports of 122427]

If yes, then we should open this bug and 1422716 then move on the CVE request/publish ossa.

@fungi, thanks for the review, it makes much sense, here is the updated impact description:

Title: Glance import task leaks image in backend
Reporter: Abhishek Kekane (NTT)
Products: Glance
Affects: 2014.2 versions through 2014.2.2

Description:
Abhishek Kekane from NTT reported a vulnerability in the Glance import task. By creating numerous images using the task API and deleting them, an authenticated attacker may accumulate untracked image data in the backend resulting in potential resource exhaustion and denial of service. All glance setups are affected.

Thierry Carrez (ttx) wrote :

Impact desc +1

tags: removed: juno-backport-potential
Changed in glance:
status: Triaged → In Progress
Thierry Carrez (ttx) wrote :

Public patches posted

information type: Private Security → Public Security

Minor:

> All glance setups are affected.

If this is specific to image import (import-from) then it only affects v2 (v1 has no import functionality).

summary: Image data remains in backend after deleting the image created using
- task api (import-from)
+ task api (import-from) (CVE-2015-1881)
Thierry Carrez (ttx) on 2015-02-23
Changed in glance:
status: In Progress → Fix Committed
Changed in ossa:
status: Triaged → In Progress
summary: - Image data remains in backend after deleting the image created using
- task api (import-from) (CVE-2015-1881)
+ [OSSA 2015-004] Image data remains in backend after deleting the image
+ created using task api (import-from) (CVE-2015-1881)
Changed in ossa:
status: In Progress → Fix Released
Thierry Carrez (ttx) on 2015-03-19
Changed in glance:
status: Fix Committed → Fix Released
Jeremy Stanley (fungi) on 2015-04-14
description: updated
Thierry Carrez (ttx) on 2015-04-30
Changed in glance:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers