[OSSA 2014-021] auth token is exposed in meter http.request (CVE-2014-4615)

Bug #1321080 reported by Zhikun Liu
266
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceilometer
Invalid
Undecided
gordon chung
Havana
Fix Released
Critical
Grant Murphy
Icehouse
Fix Committed
Critical
gordon chung
OpenStack Security Advisory
Fix Released
Medium
Tristan Cacqueray
neutron
Fix Released
Undecided
gordon chung
Icehouse
Fix Released
Undecided
Grant Murphy
oslo-incubator
Fix Released
Critical
gordon chung
Havana
Fix Committed
Critical
Grant Murphy
Icehouse
Fix Committed
Undecided
Unassigned
pycadf
Fix Released
Critical
gordon chung

Bug Description

auth token is exposed in meter http.request

# curl -i -X GET -H 'X-Auth-Token: 258ab6539b3b4eae8b3af307b8f5eadd' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-ceilometerclient' http://0.0.0.0:8777/v2/meters/http.request

-----------
snip..
{"counter_name": "http.request", "user_id": "0", "resource_id": "ip-9-37-74-33:8774", "timestamp": "2014-05-16T17:42:16.851000", "recorded_at": "2014-05-16T17:42:17.039000", "resource_metadata": {"request.CADF_EVENT:initiator:host:address": "9.44.143.6", "request.CADF_EVENT:initiator:credential:token": "4724 xxxxxxxx 8478", "request.RAW_PATH_INFO": "/v2/9af97e383dad44969bd650ebd55edfe0/servers/060c76a5-0031-430d-aa1e-01f9b3db234b", "request.REQUEST_METHOD": "DELETE", "event_type": "http.request", "request.HTTP_X_TENANT_ID": "9af97e383dad44969bd650ebd55edfe0", "request.CADF_EVENT:typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "request.HTTP_X_PROJECT_NAME": "ibm-default", "host": "nova-api", "request.SERVER_PORT": "8774", "request.REMOTE_PORT": "55258", "request.HTTP_X_USER_ID": "0", "request.HTTP_X_AUTH_TOKEN": "4724d3dd6b984079a58eecf406298478", "request.CADF_EVENT:action": "delete", "request.CADF_EVENT:target:typeURI": "service/compute/servers/server", "request.HTTP_USER_AGENT": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0",
snip...

auth token is masked in "request.CADF_EVENT:initiator:credential:token": "4724 xxxxxxxx 8478".
But it is exposed in "request.HTTP_X_AUTH_TOKEN": "4724d3dd6b984079a58eecf406298478"

Revision history for this message
gordon chung (chungg) wrote :

notifier.py grabs all environment variables. it should probably filter out HTTP_X_AUTH_TOKEN

affects: ceilometer → oslo
Changed in oslo:
assignee: nobody → gordon chung (chungg)
gordon chung (chungg)
Changed in pycadf:
assignee: nobody → gordon chung (chungg)
importance: Undecided → Critical
gordon chung (chungg)
information type: Public → Private
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Once a bug is public, we won't handle it privately. Setting it back to public

Changed in ossa:
status: New → Incomplete
information type: Private → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/94666

Changed in oslo:
status: New → In Progress
Ben Nemec (bnemec)
Changed in oslo:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/94666
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=09281ccf7837b70962ad2dfbaa1e84722ad987e8
Submitter: Jenkins
Branch: master

commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Changed in oslo:
status: In Progress → Fix Committed
Zhikun Liu (zhikunliu)
tags: added: icehouse-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/94770

Revision history for this message
Thierry Carrez (ttx) wrote : Re: auth token is exposed in meter http.request

I think this one will need an OSSA. I suspect that meter is traditionally read by people other than tenant admins ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to pycadf (master)

Fix proposed to branch: master
Review: https://review.openstack.org/94878

Changed in pycadf:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/94891

Changed in neutron:
assignee: nobody → gordon chung (chungg)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to pycadf (master)

Reviewed: https://review.openstack.org/94878
Committed: https://git.openstack.org/cgit/openstack/pycadf/commit/?id=966d4410a1a69e0a3af678442a1a965dae80d720
Submitter: Jenkins
Branch: master

commit 966d4410a1a69e0a3af678442a1a965dae80d720
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:11:52 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: I11d9f2f23fc3b60c945c33d4d02bd7640d88a083
    Closes-Bug: #1321080

Changed in pycadf:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ossa:
status: Incomplete → Confirmed
importance: Undecided → Medium
Changed in ossa:
assignee: nobody → Tristan Cacqueray (tristan-cacqueray)
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: auth token is exposed in meter http.request

It seems that this was introduced in Icehouse, thus we are still missing those patches:
* Ceilometer master and stable/icehouse
* Neutron stable/icehouse

Gordon, could you please propose Ceilometer fixes as well ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/96943

Changed in ceilometer:
assignee: nobody → gordon chung (chungg)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/96944

Revision history for this message
gordon chung (chungg) wrote : Re: auth token is exposed in meter http.request

@Tristan, thanks for letting me know. i completely forgot we had middleware module in Ceilometer.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

@Gordon, so I have a couple question for the impact description draft.

From the bug description, it appears that the leaked "request.HTTP_X_AUTH_TOKEN: 4724" is not the same than the one provided in the curl command "-H 'X-Auth-Token: 258ab"
So is the leak the token of the user requesting the notifier, or is it the admin_token defined in [filter:authtoken] configuration ?

The conditions for this leak to happen is when the notifier middleware is set after authtoken, which is not by default right ?

Revision history for this message
gordon chung (chungg) wrote :

so the leaked HTTP_X_AUTH_TOKEN value is the one in provided in curl command (i assume the description is using curl command and request object that aren't related)... it is not the admin_token defined in [filter:authtoken] configuration

you are correct that the leak happens only if notifier middleware is used after auth_token middleware (which it usually is)... by default the notifier middleware is not enabled in any service.

Zhikun Liu (zhikunliu)
tags: added: havana-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (master)

Change abandoned by gordon chung (<email address hidden>) on branch: master
Review: https://review.openstack.org/96943
Reason: this code doesn't work against master due to switch to oslo.messaging. abandoning for this bug fix: https://bugs.launchpad.net/ceilometer/+bug/1327084

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: auth token is exposed in meter http.request

@Zhi Kun Liu, Havana is impacted as well ?

@All, While oslo-incubator is not supported, should we include it in this OSSA ? Is it realistic to use this middleware out of Oslo in another service or only Neutron and Ceilometer are actually impacted ?

In the meantime, here is the impact description draft #1:

Title: User token leak to message queue in the notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron, Ceilometer, Oslo
Versions: 2014.1.1

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in Neutron and Ceilometer or through the Oslo library. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

There are 2 copies of the notifier middleware in different places in Oslo.

The copy in the incubator is used by projects that have not yet updated to oslo.messaging, such as neutron.

There is also a copy in the PyCADF library, used by projects that have updated to oslo.messaging, such as ceilometer.

Based on the history here, it looks like both copies have been fixed, so I think changing the impact description to say "the PyCADF library" instead of "the Oslo Library" will make it clear which library needs to be updated.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

@Doug, Thanks for clarifying!
Though from https://wiki.openstack.org/wiki/Security_supported_projects, oslo-incubator, oslo.messaging and PyCADF are not security supported projects (at least not in OSSA territory).

However if the notifier middleware is known to be used in services other than Neutron and Ceilometer, I'm wondering how to cover that.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote :

The incubator isn't on the OSSA list because the code in the incubator is copied into other projects that are on the list, and it's assumed that changes go into the incubator before being released into the project(s) using the modules.

oslo.messaging and PyCADF are new releases from the oslo program, and are being added to the list (probably during Juno, but that's not set for certain).

I'm not certain what uses the notifier middleware. Technically, it's middleware, so it don't have to be included in a project for a deployer to use it.

@Gordon, do you have any insight into other projects using the middleware?

Revision history for this message
gordon chung (chungg) wrote :

the original blueprint for notifier middleware is: https://blueprints.launchpad.net/ceilometer/+spec/count-api-requests. i'm unaware of anyone using the notifier middleware on its alone. to my knowledge, the main consumer of notifier middlware is pyCADF (and its audit middlware).

regarding the audit middleware:

the audit middleware (from oslo-incubator) was synced into Neutron in icehouse as a side effect of another patch (so it may not even be used). the audit middleware was also synced into Ceilometer in havana i believe (to my knowledge it's not used either as pycadf is not a requirement in Ceilometer)

the audit middleware (from pycadf) was purposely set as a requirement in Nova in icehouse and is used (it is optionally enabled by deployer). this audit middleware (from pycadf) did not exist before icehouse.

i'm not aware of any other projects pulling in pyCADF (and it's audit middleware).

hope this brain dump helps :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/94891
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bb4f44654f6765c4e1fbcf92375c273494151099
Submitter: Jenkins
Branch: master

commit bb4f44654f6765c4e1fbcf92375c273494151099
Author: Gordon Chung <email address hidden>
Date: Thu May 22 10:51:25 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Closes-Bug: #1321080
    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
Zhikun Liu (zhikunliu) wrote : Re: auth token is exposed in meter http.request

@Tristan Cacqueray, I checked nova and neutorn codes in Havana, they don't have audit and notifier middleware. So this does not impact Havana. It's only an internal problem of us. Thanks for your reminding!

tags: removed: havana-backport-potential
Thierry Carrez (ttx)
Changed in ossa:
status: Confirmed → Triaged
Revision history for this message
Thierry Carrez (ttx) wrote :

OK, this is confusing. Let me try to get an accurate picture of affected versions:

oslo-incubator contains affected code in master (patched), stable/icehouse (in review) and stable/havana
That code was copied in:

Neutron: Juno (patched), Icehouse
Ceilometer: Icehouse (in review), Havana

Then it was adopted in:
pyCADF all versions <= 0.5 (0.5.1 contains the fix)

My understanding is that oslo.messaging is not affected.

Changed in ceilometer:
status: In Progress → Invalid
Changed in pycadf:
status: Fix Committed → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote :

Adjusted tasks to match.
Here is how I would rewrite the advisory:

------
Title: User token leak to message queue in pyCADF notifier middleware
Reporter: Zhi Kun Liu (IBM)
Products: Neutron (2014.1 versions up to 2014.1.1)
          Ceilometer (2013.2 versions up to 2013.2.3, 2014.1 versions up to 2014.1.1)
          pyCADF library (all versions up to 0.5.0)

Description:
Zhi Kun Liu from IBM reported a vulnerability in the notifier middleware available in the PyCADF library and formerly copied into Neutron and Ceilometer code. An attacker with read access to the message queue may obtain authentication tokens used in REST requests (X_AUTH_TOKEN) that goes through the notifier middleware. All services using the notifier middleware configured after the auth_token middleware pipeline are impacted.
------

NB: that would mean from now on we support PyCADF, but I think now would be a good time to start.

Thierry Carrez (ttx)
Changed in neutron:
milestone: none → juno-1
status: Fix Committed → Fix Released
Changed in oslo:
milestone: none → juno-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (stable/icehouse)

Reviewed: https://review.openstack.org/94770
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Submitter: Jenkins
Branch: stable/icehouse

commit 354a9f99d177fd14d86e099ee3ffa91b9d12b5bd
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/100414

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/101097

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/icehouse)

Reviewed: https://review.openstack.org/101097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0324965a0c2987e5cad6276f011682dec184205f
Submitter: Jenkins
Branch: stable/icehouse

commit 0324965a0c2987e5cad6276f011682dec184205f
Author: Grant Murphy <email address hidden>
Date: Thu Jun 19 02:30:13 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (stable/havana)

Reviewed: https://review.openstack.org/100414
Committed: https://git.openstack.org/cgit/openstack/oslo-incubator/commit/?id=d97bd2a564cb06c613678407fd074985be73f4d5
Submitter: Jenkins
Branch: stable/havana

commit d97bd2a564cb06c613678407fd074985be73f4d5
Author: Gordon Chung <email address hidden>
Date: Tue May 20 12:30:41 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080
    (cherry picked from commit 09281ccf7837b70962ad2dfbaa1e84722ad987e8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/101799

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/havana)

Reviewed: https://review.openstack.org/101799
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=264f3b0d9640edeac743f339786e0a3b22c0f6c2
Submitter: Jenkins
Branch: stable/havana

commit 264f3b0d9640edeac743f339786e0a3b22c0f6c2
Author: Grant Murphy <email address hidden>
Date: Mon Jun 23 05:07:54 2014 +0000

    remove token from notifier middleware

    oslo-incubator sync to address the security bug
    in middleware (as below).

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

summary: - auth token is exposed in meter http.request
+ auth token is exposed in meter http.request (CVE-2014-4615)
Changed in ossa:
status: Triaged → In Progress
summary: - auth token is exposed in meter http.request (CVE-2014-4615)
+ [OSSA 2014-021] auth token is exposed in meter http.request
+ (CVE-2014-4615)
Changed in ossa:
status: In Progress → Fix Committed
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/icehouse)

Reviewed: https://review.openstack.org/96944
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2b6454f9f4e0585949ab68a91ed405755438d76e
Submitter: Jenkins
Branch: stable/icehouse

commit 2b6454f9f4e0585949ab68a91ed405755438d76e
Author: gordon chung <email address hidden>
Date: Fri May 30 17:11:18 2014 -0400

    remove token from notifier middleware

    notifier middleware is capturing token and sending it to MQ. this
    is not advisable so we should filter it out.

    Change-Id: Ia1bfa1bd24989681db1d2f385defc12e69a01f8d
    Closes-Bug: #1321080

Revision history for this message
Matthew Edmonds (edmondsw) wrote :

why is the CVE for this still not public? It still just says it has been reserved... "This candidate has been reserved by an organization or individual that will use it when announcing a new security problem. When the candidate has been publicized, the details for this candidate will be provided."

I'm guessing this was just an oversight. Can someone fix it?

Revision history for this message
Thierry Carrez (ttx) wrote :

It takes months, sometimes years for MITRE to come back to a reserved CVE and fill the appropriate information on their website. In the mean time, the CVE number serves as a reference number for all the people that need to coordinate on an issue.

Revision history for this message
Jeremy Stanley (fungi) wrote :

It was publicly assigned by MITRE in http://www.openwall.com/lists/oss-security/2014/06/24/6 and sometimes it takes their editorial board a while to compose and publish the official CVE description (can be on the order of several months).

Thierry Carrez (ttx)
Changed in neutron:
milestone: juno-1 → 2014.2
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers