[OSSA 2015-019] Image status can be changed by passing header 'x-image-meta-status' with PUT operation using v1 (CVE-2015-5251)

[Hemanth Makkapati]

Using Glance v1, one is able to change the status of an image to any one of the valid statuses by passing the header 'x-image-meta-status' with PUT on /images/<image id>. This bug provides a way for an image to transition states that are otherwise not possible in an image's lifecycle.

See http://paste.openstack.org/show/pNL7kvIZUz7cWJQwX64d/ for a reproduction of this behavior on devstack.

As shown in the above paste, though one is able to change the status of an active image to queued, uploading data after re-setting the status to queued fails with a 400[1]. Though the purpose of [1] appears to be slightly different, it's fortunately saving us from badly breaking the immutability guarantees of glance images.

[1] https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L760-L765

NOTE: Marking this as a security vulnerability for now as users would be able to activate the deactivated images on their own. This probably affects deployments only where v1 is exposed publicly. However, it's probably worth discussing this from a security perspective as well.

[Grant Murphy (gmurphy)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Nikhil Komawar (nikhil-komawar)]

This is a valid bug in Glance. It's not very insecure but, if combined with certain features of v2 like deactivation of an image (that's only allowed by admin by default), this change may result into bad image state and potentially give attackers the access to unauthorized image data.

Also, this has a per-condition that the v1 endpoint for Glance needs to be exposed. Not all deployments allow this and is not a standard, recommended practice.

[Nikhil Komawar (nikhil-komawar)]

Possible solution is to validate the headers and give back a 400 if 'x-image-meta-status' key is found in the update call.

[Erno Kuvaja (jokke)]

I'm happy to see that the immutability does not seem to be threatened currently. That gives us way more room to breath.

Even still being security issue I think this should be made public as this bug was widely discussed in weekly glance team meeting: http://eavesdrop.openstack.org/meetings/glance/2015/glance.2015-08-06-14.03.log.txt 14:29 +

Yes the image might get rendered unusable via mechanism that was never supposed to be there is quite a bit less worrying than what was discussed on the meeting.

[Erno Kuvaja (jokke)]

After having a chat with Hemanth, I'd like to pull back that previous statement about opening this bug. There is attack vector that has not been discussed openly. If admin has deactivated image due to malicious content/behavior, that image can be put back to queued and activated without approval.

This would allow booting new VMs from that potentially malicious image. So lets keep this closed for now.

[Stuart McLaren (stuart-mclaren)]

I've found a couple of ways to change the image bytes (break the immutability guarantee).

1. Set the image back to queued and upload using v2

(http://paste.openstack.org/show/412169)

2. Use --location to replace the image location

(http://paste.openstack.org/show/412172)

When set_image_location is allowed by policy it's always possible for a user to upload new
bytes to the location behind glance's back. But previously that would result in a checksum error
when the user went to download the replaced bytes. The change in behaviour here is that the
checksum, size, etc of the image are changed so the download will not raise any error.

I tried copy-from to change the image bytes, and I also tried setting the image status from the nova client.
Both failed to do anything bad.

[Hemanth Makkapati (hemanth-makkapati)]

Oh well! I didn't try uploading with v2. Good find, Stuart!

Also, mucking with my test environment further, I discovered that one can change created_at as well using the header 'x-image-meta-created_at'. I haven't tried doing the same with updated_at, deleted_at and deleted fields but I don't see anything that stops them from getting modified either. I will try it here in a bit.

A lot of it has to do with the whitelisting of headers here, https://github.com/openstack/glance/blob/master/glance/common/utils.py#L59-L70.

It appears that this was done very explicitly. Does anyone know what could be the reason for it?

[Stuart McLaren (stuart-mclaren)]

From memory, those whitelisted headers are headers that are legal to return in a response to a user.

I think the idea was to prevent the leaking of a header that we don't want to return. So, when we return
a v1 response to a user the set of 'x-image-meta-...' headers that is possible for the user to see is defined
by this whitelist. (There had been a bug where we were returning something the user shouldn't see as a
header).

[Stuart McLaren (stuart-mclaren)]

@Hemanth @Nikhil

Is this something you guys have a chance to start work on?

If not, I can try to find time to take a look.

[Hemanth Makkapati (hemanth-makkapati)]

Thanks for the note on whitelisted headers. What you said makes sense but it seems the whitelisting is used in a different context here. It's used for accepting headers in the request as opposed to returning headers in the response.

[Stuart McLaren (stuart-mclaren)]

Looking again at the whitelisting of headers. It looks like previously any 'x-image-meta-blah' header that the user provided would be passed from the api to the registry. The whitelist meant that any unexpected 'x-image-meta-xxx' headers would be rejected with 400. (But the whitelist doesn't attempt to handle different behaviour for any of the defined headers.)

In v2 an attempt to write to a 'read only' property such as status or updated_at, or a reserved property such as 'owner' will raise a 403 (Forbidden).

In v1, if an attempt to write to 'owner' is made the 'x-image-meta-owner' header will effectively just be dropped and 200 will be returned. If the image is active and an attempt is made to change size, checksum a 403 will be returned.

We'll need to decide the behaviour for attempts to update 'status', 'updated_at' etc, ie whether to silently drop them or return 403.

[Hemanth Makkapati (hemanth-makkapati)]

Stuart,
Nikhil was going to take a crack at it. But, I'm not sure of the current status. I think he is travelling currently. So, feel free to give it a go.

[Stuart McLaren (stuart-mclaren)]

Here's the current behaviour for the various headers which can be supplied:

x-image-meta-location: 400 (ok)
x-image-meta-size: 403 (ok)
x-image-meta-is_public: N/A
x-image-meta-disk_format: N/A
x-image-meta-container_format: N/A
x-image-meta-name: N/A
x-image-meta-status: 200 (bad, bug 1482371)
x-image-meta-copy_from: 200 (dropped, ok)
x-image-meta-uri: 200 (dropped, ok)
x-image-meta-checksum: 403 (ok)
x-image-meta-created_at: 200 (dropped, ok)
x-image-meta-updated_at: 200 (dropped, ok)
x-image-meta-deleted_at: 200 (dropped, ok)
x-image-meta-min_ram: N/A
x-image-meta-min_disk: N/A
x-image-meta-owner: 200 (dropped, ok)
x-image-meta-store: 200 (dropped, ok)
x-image-meta-id: 500 (bad, new bug 1483353)
x-image-meta-protected: N/A
x-image-meta-deleted: 200 (dropped, ok)
x-image-meta-virtual_size: N/A

It probably makes sense to return 403 for 'id', since the current behaviour won't update any other parameters when id is provided.

In the case of status we could consider just ignoring it, ie matching the behaviour of 'owner', ie it is not really something 'settable' in v1, so ignore it.

[Stuart McLaren (stuart-mclaren)]

However, we get a '400' in the case of an invalid status -- we should probably try not to change this.

$ curl -v -X PUT http://127.0.0.1:9292/v1/images/eda41b92-a79f-46d1-8248-dfcb5e9227ae -H 'X-Auth-Token: 7535a1be77e3459e8e4928aae02a8042' -H 'x-image-meta-id: 1234' -H 'x-image-meta-status: huh'
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 9292 (#0)
> PUT /v1/images/eda41b92-a79f-46d1-8248-dfcb5e9227ae HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 127.0.0.1:9292
> Accept: */*
> X-Auth-Token: 7535a1be77e3459e8e4928aae02a8042
> x-image-meta-id: 1234
> x-image-meta-status: huh
>
< HTTP/1.1 400 Bad Request
< Content-Length: 370
< Content-Type: text/html; charset=UTF-8
< X-Openstack-Request-Id: req-req-0a5b4e62-325e-4add-a5aa-1863eb3b5a95
< Date: Mon, 10 Aug 2015 17:43:29 GMT
<
<html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Failed to update image metadata. Got error: 400 Bad Request

The server could not comply with the request since it is either malformed or otherwise incorrect.

 Failed to update image metadata. Got error: Invalid image status 'huh' for image. <br /><br />

[Stuart McLaren (stuart-mclaren)]

A reference to nova code setting x-image-meta-status:

https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L184

[Flavio Percoco (flaper87)]

I think the order of validation should be:

1) Check whether the header being passed is valid/allowed.
2) Check if the value provided is valid.

The reason I believe the above is correct s that doing it the other way around we'd give the attacker hints of what status are valid. This might not be such a big issue since image's status are public anyway but still, I believe that would be the right workflow.

There's no point on letting the user guess the right status to then let the user know that status can't be updated.

Hope the above makes sense.

[Stuart McLaren (stuart-mclaren)]

Here's a first pass at a patch.

There's probably no perfect solution here. This is hopefully someways backward compatible/sane.

[Flavio Percoco (flaper87)]

As mentioned in my previous comment, I think it'd be better to raise 403 in every case as a general thing. I believe we should do that in v2, at least. Nonetheless, I think the current patch is good and it preserves the current behavior on v1, which is in maintenance mode.

+1 for the patch, tested locally. Thanks, Stuart.

[Brian Rosmaita (brian-rosmaita)]

Stuart, the patch looks good to me, too. I don't see a way to disallow 'setting' to the same status without causing major problems ... that was a good catch to notice nova setting status as part of snapshotting on xenserver.

[Erno Kuvaja (jokke)]

Stuart, Tristan, Grant,

I think we have fix we can agree, any progress on publication schedule?

We're two weeks from the OSSA confirmation and Wednesday 2 weeks from patch agreement.

[Grant Murphy (gmurphy)]

Please review this impact description for correctness. I will use it to request a CVE for this issue once I've verified that it is accurate.

Title: Glance v1 API image status manipulation
Reporter: Hemanth Makkapati (Rackspace)
Products: Glance
Affects: all 2013.2 and 2014.1 versions, 2014.2 versions through 2014.2.3,
          and 2015.1 versions through 2015.1.1

Description:
Hemanth Makkapati of Rackspace reported a vulnerability in Glance. By submitting
a HTTP PUT request with a 'x-image-meta-status' header, a malicious tenant can
manipulate the status of public images without requiring administrative
privileges. This may allow an attacker to reactivate a malicious images that
was disabled by an administrator, potentially impacting other tenants. Only setups
using the Glance v1 API are affected by this flaw.

[Erno Kuvaja (jokke)]

Hi Grant,

Thanks for that. As Stuart pointed out on comment #6 it would be also possible to upload replacement data utilizing v2 API after the initial status change over v1 API. This would break the immutability promise we have.

Rest, please correct me, but this issue was only affecting images owned by tenant (and admins), right? One couldn't alter images they did not own.

[Stuart McLaren (stuart-mclaren)]

@Grant

Some tweaks:

"By submitting a HTTP PUT request with a 'x-image-meta-status' header, a malicious tenant can
manipulate the status of public images without requiring administrative
privileges."

This isn't specific to public images. Also, the admin privilege is a bit of a red herring I think. Basically, standard users can only modify the state of images that they own (as Erno says above).

Maybe something like:

"By submitting a HTTP PUT request with a 'x-image-meta-status' header, a tenant can
manipulate the status of their images. In some cases this then allows replacing the image contents by reuploading them -- which should not be allowed as 'active' images are considered immutable."

"Only setups using the Glance v1 API are affected by this flaw."

"Only setups using the Glance v1 API allow the illegal modification of image status. Only setups which also use the v2 API may allow a subsequent re-upload of image contents."

[Jeremy Stanley (fungi)]

Given a tenant can't actually alter images which aren't under their control, what are the exploit scenarios for this vulnerability? Simply reenabling images of theirs which the admin has disabled, or substituting a malicious replacement image after conclusion of an audit validating the image being surreptitiously replaced?

[Erno Kuvaja (jokke)]

Jeremy,

Yes, both of those scenarios are valid.

[Erno Kuvaja (jokke)]

Lets say, cloud has workflow for 3rd party images where the image gets uploaded by 3rd party, validated and marked public by cloud admin. With this bug the 3rd party can replace the image (and it's checksum) after it has been made public providing for example malicious payload breaking the immutability promise glance has. This would need V1 & V2 APIs.

The other possibility is that private/shared/public image gets flagged malicious and disabled for further analysis. Tenant can move it back from being disabled to active and allow booting from it again. Only V1 API is needed for such actions.

[Stuart McLaren (stuart-mclaren)]

You can also work around the storage usage limit by toggling images between 'killed' and 'active'. ie toggle them to 'killed' so you won't be 1) billed for them 2) they wont count towards your storage useage; then toggle back to active before you want to boot them.

[Jeremy Stanley (fungi)]

Toggling images back from the dead seems like an iffy exploit scenario, since I would expect service providers to delete these in an effort to reclaim space anyway. The surreptitious replacement however, if it's documented that image-to-UUID mapping is immutable, does at least seem like a failing in a security guarantee users might depend on.

[Brian Rosmaita (brian-rosmaita)]

I want to stress the point Hemanth made somewhere above about image activate/deactivate.

In Kilo, Glance added a security feature that allows an admin to "deactivate" an image while a deployer investigates it. (The image still exists, its record is still visible, but Glance will refuse to download the image data, and hence instances can't be booted from it.) An image owner with access to the v1 API can use this exploit to put the image back into active status, which pretty much negates the deactivation security feature.

[Tristan Cacqueray (tristan-cacqueray)]

If standard users can only modify the state of images that they own, whenever an admin deactivate an image, what would prevent the user to re-upload it instead of switching the status of the deactivated image back to active ?

[Jeremy Stanley (fungi)]

Also if an admin deactivates the image and then the user reactivates it through this bug, can't the admin just delete the image (after making a copy for themselves) or, for that matter, disable the (now obviously abusive) tenant entirely?

[Grant Murphy (gmurphy)]

Ok. How about this? Cover all the cases above?

Title: Glance v1 API image status manipulation
Reporter: Hemanth Makkapati (Rackspace)
Products: Glance
Affects: all 2013.2 and 2014.1 versions, 2014.2 versions through 2014.2.3,
          and 2015.1 versions through 2015.1.1

Description:
Hemanth Makkapati of Rackspace reported a vulnerability in Glance. By submitting a HTTP PUT request with a 'x-image-meta-status' header, a tenant can manipulate the status of their images. A malicious tenant may exploit this flaw to reactivate disabled images, bypass storage quotas and in some cases replace image contents. Setups using the Glance v1 API allow the illegal modification of image status. Setups which also use the v2 API may allow a subsequent re-upload of image contents.

[Jeremy Stanley (fungi)]

Grant's updated impact description in comment #32 looks great.

[Brian Rosmaita (brian-rosmaita)]

I agree, Grant's impact description in comment #32 lgtm.

[Grant Murphy (gmurphy)]

Thanks. CVE requested.

[Grant Murphy (gmurphy)]

Can someone take another look at the patch for this and organize backports as required? I will send out the pre-OSSA and set the disclosure date once we have that. Thanks.

[Flavio Percoco (flaper87)]

Grant, I just took another look at the patch. Looks fine. I'm happy to help w/ the backports. Let me know when it'll be schedule. I'm in CEST.

[Flavio Percoco (flaper87)]

I've uploaded the backports for kilo and juno and I've also re-generated the one for liberty using git-tools.

Please, do verify them and let me know. These patches are the ones the will be published when this bug will be disclosed

[Tristan Cacqueray (tristan-cacqueray)]

Patch applies correctly and tests seems ok. I confirm this fixed the issue in stable/kilo.

Can we get another +2 from glance-coresec ?

[Stuart McLaren (stuart-mclaren)]

+2

Thanks Flavio!

There is at least one operator who has deployed a version of this patch successfully.

[Jeremy Stanley (fungi)]

To reiterate the boilerplate in the bug description, please avoid (even private) disclosure of the vulnerability and associated patches to other individuals not already approved for access to this information. Deploying the patches in production in a large service provider runs the risk that others within the same organization have access to the details of this fix in advance of the downstream pre-notification.

[Erno Kuvaja (jokke)]

+2 looks good to go on all branches. ( glance-core ! glance-coresec)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/226337

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/226338

[OpenStack Infra (hudson-openstack)]

Change abandoned by Flavio Percoco (<email address hidden>) on branch: master
Review: https://review.openstack.org/226355
Reason: https://review.openstack.org/#/q/I44fadf32abb57c962b67467091c3f51c1ccc25e6,n,z

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226336
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=34f9f037136061f7437761e86ea9f0864b297619
Submitter: Jenkins
Branch: master

commit 34f9f037136061f7437761e86ea9f0864b297619
Author: Stuart McLaren <email address hidden>
Date: Tue Aug 11 10:37:09 2015 +0000

    Prevent image status being directly modified via v1

    Users shouldn't be able to change an image's status directly via the
    v1 API.

    Some existing consumers of Glance set the x-image-meta-status header in
    requests to the Glance API, eg:

    https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L184

    We should try to prevent users setting 'status' via v1, but without breaking
    existing benign API calls such as these.

    I've adopted the following approach (which has some prior art in 'protected properties').

    If a PUT request is received which contains an x-image-meta-status header:

    * The user provided status is ignored if it matches the current image
      status (this prevents benign calls such as the nova one above from
      breaking). The usual code (eg 200) will be returned.

    * If the user provided status doesn't match the current image status (ie
      there is a real attempt to change the value) 403 will be returned. This
      will break any calls which currently intentionally change the status.

    APIImpact

    Closes-bug: 1482371

    Change-Id: I44fadf32abb57c962b67467091c3f51c1ccc25e6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226343
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=1a4a56ab6698c274c64517d6e3f9cf1a045609b6
Submitter: Jenkins
Branch: master

commit 1a4a56ab6698c274c64517d6e3f9cf1a045609b6
Author: Grant Murphy <email address hidden>
Date: Tue Sep 22 07:20:00 2015 -0700

    Adds OSSA-2015-019

    Related-Bug: #1482371
    Change-Id: I7d5fe8c6b3fd5c0936af809ed7cf30247c966a9b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226337
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=9beca533f42ae1fc87418de0c360e19bc59b24b5
Submitter: Jenkins
Branch: stable/kilo

commit 9beca533f42ae1fc87418de0c360e19bc59b24b5
Author: Stuart McLaren <email address hidden>
Date: Tue Aug 11 10:37:09 2015 +0000

    Prevent image status being directly modified via v1

    Users shouldn't be able to change an image's status directly via the
    v1 API.

    Some existing consumers of Glance set the x-image-meta-status header in
    requests to the Glance API, eg:

    https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L184

    We should try to prevent users setting 'status' via v1, but without breaking
    existing benign API calls such as these.

    I've adopted the following approach (which has some prior art in 'protected properties').

    If a PUT request is received which contains an x-image-meta-status header:

    * The user provided status is ignored if it matches the current image
      status (this prevents benign calls such as the nova one above from
      breaking). The usual code (eg 200) will be returned.

    * If the user provided status doesn't match the current image status (ie
      there is a real attempt to change the value) 403 will be returned. This
      will break any calls which currently intentionally change the status.

    APIImpact

    Closes-bug: 1482371

    Change-Id: I44fadf32abb57c962b67467091c3f51c1ccc25e6
    (cherry picked from commit 4d08db5b6d42323ac1958ef3b7417d875e7bea8c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226338
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=45be8e1c620c50f3cbca76f561945200a8843bc8
Submitter: Jenkins
Branch: stable/juno

commit 45be8e1c620c50f3cbca76f561945200a8843bc8
Author: Stuart McLaren <email address hidden>
Date: Tue Aug 11 10:37:09 2015 +0000

    Prevent image status being directly modified via v1

    Users shouldn't be able to change an image's status directly via the
    v1 API.

    Some existing consumers of Glance set the x-image-meta-status header in
    requests to the Glance API, eg:

    https://github.com/openstack/nova/blob/master/plugins/xenserver/xenapi/etc/xapi.d/plugins/glance#L184

    We should try to prevent users setting 'status' via v1, but without breaking
    existing benign API calls such as these.

    I've adopted the following approach (which has some prior art in 'protected properties').

    If a PUT request is received which contains an x-image-meta-status header:

    * The user provided status is ignored if it matches the current image
      status (this prevents benign calls such as the nova one above from
      breaking). The usual code (eg 200) will be returned.

    * If the user provided status doesn't match the current image status (ie
      there is a real attempt to change the value) 403 will be returned. This
      will break any calls which currently intentionally change the status.

    APIImpact

    Closes-bug: 1482371

    Change-Id: I44fadf32abb57c962b67467091c3f51c1ccc25e6
    (cherry picked from commit 4d08db5b6d42323ac1958ef3b7417d875e7bea8c)
    (cherry picked from commit 9beca533f42ae1fc87418de0c360e19bc59b24b5)