[OSSA 2014-021] auth token is exposed in meter http.request (CVE-2014-4615)

Bug #1321080 reported by Zhikun Liu on 2014-05-20
266
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Undecided
gordon chung
Havana
Fix Released
Critical
Grant Murphy
Icehouse
Fix Committed
Critical
gordon chung
OpenStack Security Advisory
Medium
Tristan Cacqueray
neutron
Undecided
gordon chung
Icehouse
Undecided
Grant Murphy
oslo-incubator
Critical
gordon chung
Havana
Critical
Grant Murphy
Icehouse
Undecided
Unassigned
pycadf
Critical
gordon chung

Bug Description

auth token is exposed in meter http.request

# curl -i -X GET -H 'X-Auth-Token: 258ab6539b3b4eae8b3af307b8f5eadd' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-ceilometerclient' http://0.0.0.0:8777/v2/meters/http.request

-----------
snip..
{"counter_name": "http.request", "user_id": "0", "resource_id": "ip-9-37-74-33:8774", "timestamp": "2014-05-16T17:42:16.851000", "recorded_at": "2014-05-16T17:42:17.039000", "resource_metadata": {"request.CADF_EVENT:initiator:host:address": "9.44.143.6", "request.CADF_EVENT:initiator:credential:token": "4724 xxxxxxxx 8478", "request.RAW_PATH_INFO": "/v2/9af97e383dad44969bd650ebd55edfe0/servers/060c76a5-0031-430d-aa1e-01f9b3db234b", "request.REQUEST_METHOD": "DELETE", "event_type": "http.request", "request.HTTP_X_TENANT_ID": "9af97e383dad44969bd650ebd55edfe0", "request.CADF_EVENT:typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "request.HTTP_X_PROJECT_NAME": "ibm-default", "host": "nova-api", "request.SERVER_PORT": "8774", "request.REMOTE_PORT": "55258", "request.HTTP_X_USER_ID": "0", "request.HTTP_X_AUTH_TOKEN": "4724d3dd6b984079a58eecf406298478", "request.CADF_EVENT:action": "delete", "request.CADF_EVENT:target:typeURI": "service/compute/servers/server", "request.HTTP_USER_AGENT": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
snip...

auth token is masked in "request.CADF_EVENT:initiator:credential:token": "4724 xxxxxxxx 8478".
But it is exposed in "request.HTTP_X_AUTH_TOKEN": "4724d3dd6b984079a58eecf406298478"

gordon chung (chungg) wrote :

notifier.py grabs all environment variables. it should probably filter out HTTP_X_AUTH_TOKEN

affects: ceilometer → oslo
Changed in oslo:
assignee: nobody → gordon chung (chungg)
gordon chung (chungg) on 2014-05-20
Changed in pycadf:
assignee: nobody → gordon chung (chungg)
importance: Undecided → Critical
gordon chung (chungg) on 2014-05-20
information type: Public → Private

Once a bug is public, we won't handle it privately. Setting it back to public

Changed in ossa:
status: New → Incomplete
information type: Private → Public Security

Fix proposed to branch: master
Review: https://review.openstack.org/94666

Changed in oslo:
status: New → In Progress
Ben Nemec (bnemec) on 2014-05-21
Changed in oslo:
importance: Undecided → Critical

Reviewed: https://review.openstack.org/94666
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=09281ccf7837b70962ad2dfbaa1e84722ad987e8
Submitter: Jenkins
Branch: master

commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Changed in oslo:
status: In Progress → Fix Committed
Zhikun Liu (zhikunliu) on 2014-05-22
tags: added: icehouse-backport-potential

I think this one will need an OSSA. I suspect that meter is traditionally read by people other than tenant admins ?

Fix proposed to branch: master
Review: https://review.openstack.org/94878

Changed in pycadf:
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/94891

Changed in neutron:
assignee: nobody → gordon chung (chungg)
status: New → In Progress

Reviewed: https://review.openstack.org/94878
Committed: https://git.openstack.org/cgit/openstack/pycadf/commit/?id=966d4410a1a69e0a3af678442a1a965dae80d720
Submitter: Jenkins
Branch: master

commit 966d4410a1a69e0a3af678442a1a965dae80d720
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:11:52 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: I11d9f2f23fc3b60c945c33d4d02bd7640d88a083
    Closes-Bug: #1321080

Changed in pycadf:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-05-26
Changed in ossa:
status: Incomplete → Confirmed
importance: Undecided → Medium
Changed in ossa:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)

It seems that this was introduced in Icehouse, thus we are still missing those patches:
* Ceilometer master and stable/icehouse
* Neutron stable/icehouse

Gordon, could you please propose Ceilometer fixes as well ?

Fix proposed to branch: master
Review: https://review.openstack.org/96943

Changed in ceilometer:
assignee: nobody → gordon chung (chungg)
status: New → In Progress

@Tristan, thanks for letting me know. i completely forgot we had middleware module in Ceilometer.

@Gordon, so I have a couple question for the impact description draft.

From the bug description, it appears that the leaked "request.HTTP_X_AUTH_TOKEN: 4724" is not the same than the one provided in the curl command "-H 'X-Auth-Token: 258ab"
So is the leak the token of the user requesting the notifier, or is it the admin_token defined in [filter:authtoken] configuration ?

The conditions for this leak to happen is when the notifier middleware is set after authtoken, which is not by default right ?

gordon chung (chungg) wrote :

so the leaked HTTP_X_AUTH_TOKEN value is the one in provided in curl command (i assume the description is using curl command and request object that aren't related)... it is not the admin_token defined in [filter:authtoken] configuration

you are correct that the leak happens only if notifier middleware is used after auth_token middleware (which it usually is)... by default the notifier middleware is not enabled in any service.

Zhikun Liu (zhikunliu) on 2014-06-06
tags: added: havana-backport-potential

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/96943
Reason: this code doesn't work against master due to switch to oslo.messaging. abandoning for this bug fix: https://bugs.launchpad.net/ceilometer/+bug/1327084

@Zhi Kun Liu, Havana is impacted as well ?

@All, While oslo-incubator is not supported, should we include it in this OSSA ? Is it realistic to use this middleware out of Oslo in another service or only Neutron and Ceilometer are actually impacted ?

In the meantime, here is the impact description draft #1:

Title: User token leak to message queue in the notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron, Ceilometer, Oslo
Versions: 2014.1.1

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in Neutron and Ceilometer or through the Oslo library. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.

Doug Hellmann (doug-hellmann) wrote :

There are 2 copies of the notifier middleware in different places in Oslo.

The copy in the incubator is used by projects that have not yet updated to oslo.messaging, such as neutron.

There is also a copy in the PyCADF library, used by projects that have updated to oslo.messaging, such as ceilometer.

Based on the history here, it looks like both copies have been fixed, so I think changing the impact description to say "the PyCADF library" instead of "the Oslo Library" will make it clear which library needs to be updated.

@Doug, Thanks for clarifying!
Though from https://wiki.openstack.org/wiki/Security_supported_projects, oslo-incubator, oslo.messaging and PyCADF are not security supported projects (at least not in OSSA territory).

However if the notifier middleware is known to be used in services other than Neutron and Ceilometer, I'm wondering how to cover that.

Doug Hellmann (doug-hellmann) wrote :

The incubator isn't on the OSSA list because the code in the incubator is copied into other projects that are on the list, and it's assumed that changes go into the incubator before being released into the project(s) using the modules.

oslo.messaging and PyCADF are new releases from the oslo program, and are being added to the list (probably during Juno, but that's not set for certain).

I'm not certain what uses the notifier middleware. Technically, it's middleware, so it don't have to be included in a project for a deployer to use it.

@Gordon, do you have any insight into other projects using the middleware?

gordon chung (chungg) wrote :

the original blueprint for notifier middleware is: https://blueprints.launchpad.net/ceilometer/+spec/count-api-requests. i'm unaware of anyone using the notifier middleware on its alone. to my knowledge, the main consumer of notifier middlware is pyCADF (and its audit middlware).

regarding the audit middleware:

the audit middleware (from oslo-incubator) was synced into Neutron in icehouse as a side effect of another patch (so it may not even be used). the audit middleware was also synced into Ceilometer in havana i believe (to my knowledge it's not used either as pycadf is not a requirement in Ceilometer)

the audit middleware (from pycadf) was purposely set as a requirement in Nova in icehouse and is used (it is optionally enabled by deployer). this audit middleware (from pycadf) did not exist before icehouse.

i'm not aware of any other projects pulling in pyCADF (and it's audit middleware).

hope this brain dump helps :)

Reviewed: https://review.openstack.org/94891
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb4f44654f6765c4e1fbcf92375c273494151099
Submitter: Jenkins
Branch: master

commit bb4f44654f6765c4e1fbcf92375c273494151099
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:51:25 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Closes-Bug: #1321080
    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d

Changed in neutron:
status: In Progress → Fix Committed

@Tristan Cacqueray, I checked nova and neutorn codes in Havana, they don't have audit and notifier middleware. So this does not impact Havana. It's only an internal problem of us. Thanks for your reminding!

tags: removed: havana-backport-potential
Thierry Carrez (ttx) on 2014-06-09
Changed in ossa:
status: Confirmed → Triaged
Thierry Carrez (ttx) wrote :

OK, this is confusing. Let me try to get an accurate picture of affected versions:

oslo-incubator contains affected code in master (patched), stable/icehouse (in review) and stable/havana
That code was copied in:

Neutron: Juno (patched), Icehouse
Ceilometer: Icehouse (in review), Havana

Then it was adopted in:
pyCADF all versions <= 0.5 (0.5.1 contains the fix)

My understanding is that oslo.messaging is not affected.

Changed in ceilometer:
status: In Progress → Invalid
Changed in pycadf:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) wrote :

Adjusted tasks to match.
Here is how I would rewrite the advisory:

------
Title: User token leak to message queue in pyCADF notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron (2014.1 versions up to 2014.1.1)
          Ceilometer (2013.2 versions up to 2013.2.3, 2014.1 versions up to 2014.1.1)
          pyCADF library (all versions up to 0.5.0)

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in the PyCADF library and formerly copied into Neutron and Ceilometer code. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.
------

NB: that would mean from now on we support PyCADF, but I think now would be a good time to start.

Thierry Carrez (ttx) on 2014-06-12
Changed in neutron:
milestone: none → juno-1
status: Fix Committed → Fix Released
Changed in oslo:
milestone: none → juno-1
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/94770
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Submitter: Jenkins
Branch: stable/icehouse

commit 354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Reviewed: https://review.openstack.org/101097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0324965a0c2987e5cad6276f011682dec184205f
Submitter: Jenkins
Branch: stable/icehouse

commit 0324965a0c2987e5cad6276f011682dec184205f
Author: Grant Murphy <email address hidden>
Date: Thu Jun 19 02:30:13 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Reviewed: https://review.openstack.org/100414
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d97bd2a564cb06c613678407fd074985be73f4d5
Submitter: Jenkins
Branch: stable/havana

commit d97bd2a564cb06c613678407fd074985be73f4d5
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Reviewed: https://review.openstack.org/101799
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=264f3b0d9640edeac743f339786e0a3b22c0f6c2
Submitter: Jenkins
Branch: stable/havana

commit 264f3b0d9640edeac743f339786e0a3b22c0f6c2
Author: Grant Murphy <email address hidden>
Date: Mon Jun 23 05:07:54 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

summary: - auth token is exposed in meter http.request
+ auth token is exposed in meter http.request (CVE-2014-4615)
Changed in ossa:
status: Triaged → In Progress
summary: - auth token is exposed in meter http.request (CVE-2014-4615)
+ [OSSA 2014-021] auth token is exposed in meter http.request
+ (CVE-2014-4615)
Changed in ossa:
status: In Progress → Fix Committed
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/96944
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2b6454f9f4e0585949ab68a91ed405755438d76e
Submitter: Jenkins
Branch: stable/icehouse

commit 2b6454f9f4e0585949ab68a91ed405755438d76e
Author: gordon chung <email address hidden>
Date: Fri May 30 17:11:18 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Matthew Edmonds (edmondsw) wrote :

why is the CVE for this still not public? It still just says it has been reserved... "This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided."

I'm guessing this was just an oversight. Can someone fix it?

Thierry Carrez (ttx) wrote :

It takes months, sometimes years for MITRE to come back to a reserved CVE and fill the appropriate information on their website. In the mean time, the CVE number serves as a reference number for all the people that need to coordinate on an issue.

Jeremy Stanley (fungi) wrote :

It was publicly assigned by MITRE in http://www.openwall.com/lists/oss-security/2014/06/24/6 and sometimes it takes their editorial board a while to compose and publish the official CVE description (can be on the order of several months).

Thierry Carrez (ttx) on 2014-10-16
Changed in neutron:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers