[OSSA-2017-001] CatchErrors leaks sensitive values in oslo.middleware (CVE-2017-2592)

[Divya K Konoor]

I had reported LP bug https://bugs.launchpad.net/keystonemiddleware/+bug/1627696 yesterday and I see that in cases where an error of this kind occurs the auth token used to place the rest call to neutron us logged as part of the stacktrace (which logs the headers including the token). I am not sure if this needs to be handled at the oslo_middleware layer or keystonemiddleware layer.

Stacktrace from neutron:

X-Auth-Token: gAAAAABX6NfMz4Lj4sYIDHu0eXr9oxymDrJTDOOrKztp0NElSiZcs9Umr-v8P-s8VP_lz_aVKPobfoj1ROP9X9amp8ACqwa4FNRvFX5IatzwmjAKR42AZZnuD4jxoJoC05iT-UKIY81gqHsOY8v7DbqTLSE2eOFwrFKZIMQBUDlDaeqwpce0LDp-dZrM2JIta9tOz99aOH5CShyu-ihMy3F87CN3cMdK5qHIr7oM1UiXc97zgzbDOTA
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 38, in __call__
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2016-09-26 05:29:36.804 28288 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Jamie Lennox (jamielennox)]

So we should definitely fix the OSError thrown by auth_token middleware however this is going to happen with anything that gets caught by catch_errors. By printing the webob request the whole request is going to be printed into the log including headers. I'm not sure what happens here with large binary data in the request.

There are two ways we can fix this.

Either we change the logging to log only the url and basic data, or we scrub the sensitive data from the full output. Having never used the middleware I don't know which is more useful for people.

I'm attaching a patch that would scrub the token data from the logged request. We would want to scrub at least X-Auth-Token, X-Service-Token and X-Subject-Token. I don't mind if we log a simpler version of the request instead.

IMO this bug doesn't need to be private.

[Matthew Edmonds (edmondsw)]

In the past we've said that secrets logged by default (and this goes to ERROR level, so it qualifies) are vulnerabilities. See https://bugs.launchpad.net/ossa/+bug/1543402 for at least one discussion of that. That example was made public, but I think it was only because the information was already out there.

As for the fix, should we use oslo_utils.strutils.mask_password? The patterns it's currently using wouldn't match this case today, but they could (and probably should?) be updated to match.

[Steve Martinelli (stevemar)]

Marking keystonemiddleware as invalid based on Jamie's diagnosis. Also ++ to using oslo.utils

[Matthew Edmonds (edmondsw)]

I believe https://review.openstack.org/#/c/385169/ and https://review.openstack.org/#/c/385197/ should, when merged, allow oslo_utils.strutils.mask_password to be used here. Modified Jamie's patch to make use of that standard utility.

[Matthew Edmonds (edmondsw)]

The reviews I mentioned in comment 5 are essentially dead. It's been agreed that mask_password is trying to do too much for too many different cases and getting into trouble doing that. It should be used less, not more. We should proceed with Jamie's patch here.

[Steve Martinelli (stevemar)]

Let's go with Jamie's solution, we probably need an OSSSA, i think this is a Class A security bug: https://security.openstack.org/vmt-process.html

[Jeremy Stanley (fungi)]

Proposed impact description...

Title: CatchErrors leaks sensitive values in oslo.middleware
Reporter: Divya K Konoor (IBM)
Products: oslo.middleware
Affects: <=2.8.0, >=3.0.0 <=3.8.0, >=3.9.0 <=3.19.0, ==3.20.0

Description:
Divya K Konoor with IBM reported a vulnerability in oslo.middleware.
Software using the CatchError class may include sensitive values in
Tracebacks resulting in their disclosure, for example tokens handled
by keystonemiddleware leaking into neutron error logs.

[Morgan Fainberg (mdrnstm)]

The Impact statement in #8 looks good to me as long as the Affected Versions is correct.

I would potentially include that the tracebacks can include the entire request not just "some sensitive values". I would be ok with it as is however.

[Matthew Edmonds (edmondsw)]

There's probably some standard way of deciding how to fill the "Affects" line, which this probably meets, but it seems odd. Isn't the value in comment 8 equivalent to "<=2.8.0, >=3.0.0 <=3.20.0", unless there are 3.8.x and 3.19.x versions that aren't affected? And I don't see how that would be possible.

I would be ok with the description in comment 8, but something like the following might be better:

Divya K Konoor with IBM reported a vulnerability in oslo.middleware. Software using the CatchError class may include sensitive values in the error message accompanying a Traceback, resulting in their disclosure. For example keystone tokens included in API request headers may leak into neutron error logs.

[Matthew Edmonds (edmondsw)]

oh, 2.8.0, 3.8.0, and 3.19.0 were the upper constraints for liberty, mitaka and newton, respectively. So I take it that the affects line as written is assuming this will get backported to 2.8.1, 3.8.1, and 3.19.1. Makes sense.

[Divya K Konoor (dikonoor)]

What are the next steps towards making this fix available in Ocata ?

[Jeremy Stanley (fungi)]

Since it sounds like there's no dissent against Jamie's patch in comment #2, we need it backported to stable/newton and stable/mitaka branches (can omit stable/liberty now that it's past EOL). Is anyone formulating these backports yet?

[Jeremy Stanley (fungi)]

Updated impact description taking into account Liberty EOL, newer master branch releases, and Morgan's and Matthew's suggestions from comments #9 and #10...

Title: CatchErrors leaks sensitive values in oslo.middleware
Reporter: Divya K Konoor (IBM)
Products: oslo.middleware
Affects: <=3.8.0, >=3.9.0 <=3.19.0, >=3.20.0 <=3.22.0

Description:
Divya K Konoor with IBM reported a vulnerability in oslo.middleware. Software using the CatchError class may include sensitive values in the error message accompanying a Traceback, resulting in their disclosure. For example, complete API requests (including keystone tokens in their headers) may leak into neutron error logs.

[Jamie Lennox (jamielennox)]

Patch in comment #2 applies cleanly to both stable/newton and stable/mitaka.

[Divya K Konoor (dikonoor)]

Considering #15, can this patch/fix be released? Is there anything more that needs to be done?

[Jeremy Stanley (fungi)]

It looks like we missed subscribing oslo-coresec reviewers when this was first shifted away from keystone-coresec jurisdiction. I've added them now so they can provide some official pre-disclosure feedback on the proposed patch in comment #2 and proposed impact description in comment #14 before the VMT requests a CVE assignment.

[Steve Martinelli (stevemar)]

Any update on this from the oslo-coresec folks? not sure we have the runway to get this into M, N and O now. Since we're at non-client freeze this week.

[Joshua Harlow (harlowja)]

Patch from Jamie seems ok with me.

[Morgan Fainberg (mdrnstm)]

@Steve,

I think we can get this into M, N, and O. Since this is a Class A vuln (with a Pending OSSA), it should be within scope of adding and potentially cutting another release if needed / freeze exception (I'd advocate for that for most security fixes)

Once we have the ok from oslo-coresec on the impact statement (#14) we can move forward with CVE request and move towards disclosure and submitting the patch to gerrit.

@Joshua, any additions/changes/concerns with the impact statement?

[Joshua Harlow (harlowja)]

The statement in #14 seems ok to me, thanks for writing it up guys/gals :)

[Morgan Fainberg (mdrnstm)]

I have subscribed the release team as we are very close to a release and this will happen post release/freeze for oslo.middleware. It may be worth just adding a subsequent point release as soon as the changes merge.

[Divya K Konoor (dikonoor)]

Jeremy/Morgan, what's the next step in the process to release the fix ? Issuing CVE?

[Jeremy Stanley (fungi)]

Divya: Yes, a CVE was requested in private from our CNA yesterday shortly following Joshua's review of the proposed patch, and I updated the bug title earlier today once I received it. Next step is to propose a coordinated disclosure date and then send a pre-OSSA to our downstream stakeholders providing them with advance copies of the impact description and patch.

[Jeremy Stanley (fungi)]

Based on our usual notification timeline ( https://security.openstack.org/vmt-process.html#embargoed-disclosure ) I propose we send a pre-OSSA within the next 19 hours with a target coordinated disclosure time of 15:00 UTC on Wednesday, January 25. Any objections? Will any Oslo core reviewers be around to quickly usher the corresponding changes through code review?

[Jeremy Stanley (fungi)]

Given lack of feedback before the deadline, I'm revising my proposal to having a pre-OSSA sent by 15:00 UTC Monday with a target disclosure time of 15:00 UTC Thursday, January 26. Any objections? Will any Oslo core reviewers be around to quickly usher the corresponding changes through code review at that time on Thursday?

[Joshua Harlow (harlowja)]

I will be around, dims and doug are usually around also.

[Jeremy Stanley (fungi)]

Also, the impact description will be updated to the following to account for another oslo.messaging ocata release:

Affects: <=3.8.0, >=3.9.0 <=3.19.0, >=3.20.0 <=3.23.0

[Jeremy Stanley (fungi)]

Thanks Josh! I'll go ahead and get the pre-OSSA rolling with Thursday as our disclosure target in that case.

[Jeremy Stanley (fungi)]

Looks like at some point the patch from comment #2 ceased applying cleanly to the master branch, so I have rebased it (please review). I'm fairly confident it's a straightforward rebase so I'm going to go ahead and include it in the pre-OSSA I'm sending now. If it ends up needing adjustment we can do that on the fly since it's just the patch for the development branch anyway.

[Joshua Harlow (harlowja)]

Looks like a straightforward rebase to me, thanks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/425741

[Jeremy Stanley (fungi)]

As Matthew Edmonds pointed out in a comment at https://review.openstack.org/425732 this was partially (perhaps completely?) fixed in master already under separate bug 1646254 which was not identified as a vulnerability.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425741
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=0b074f5c166f091fd0bd62cc4330047ba9dfb4c6
Submitter: Jenkins
Branch: master

commit 0b074f5c166f091fd0bd62cc4330047ba9dfb4c6
Author: Jeremy Stanley <email address hidden>
Date: Thu Jan 26 14:55:39 2017 +0000

    OSSA-2017-001 (CVE-2017-2592)

    CatchErrors leaks sensitive values in oslo.middleware

    Change-Id: I2a85e96f457e58cc7f2160d733bdc7b1fe8de3df
    Closes-Bug: #1628031

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.middleware 3.23.1 release.

[Jeremy Stanley (fungi)]

Due to a configuration error our code review system didn't link to the fixes here in the bug report, so for posterity:

    https://review.openstack.org/425734 (Mitaka)
    https://review.openstack.org/425732 (Newton)
    https://review.openstack.org/425730 (Ocata)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425732
Committed: https://git.openstack.org/cgit/openstack/oslo.middleware/commit/?id=6c0f50c1f5f4122b31dbfe25aacdce596bf4b648
Submitter: Jenkins
Branch: stable/newton

commit 6c0f50c1f5f4122b31dbfe25aacdce596bf4b648
Author: Jamie Lennox <email address hidden>
Date: Wed Sep 28 15:03:53 2016 +1000

    Filter token data out of catch_errors middleware

    If an exception is caught by the catch_errors middleware the entire
    request is dumped into the log including sensitive information like
    tokens. Filter that information before outputting the failed request.

    Closes-Bug: #1628031
    Change-Id: I2563403993513c37751576223275350cac2e0937

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425734
Committed: https://git.openstack.org/cgit/openstack/oslo.middleware/commit/?id=ec073669a49267abcb0c1d776b9050342dac5a4a
Submitter: Jenkins
Branch: stable/mitaka

commit ec073669a49267abcb0c1d776b9050342dac5a4a
Author: Jamie Lennox <email address hidden>
Date: Wed Sep 28 15:03:53 2016 +1000

    Filter token data out of catch_errors middleware

    If an exception is caught by the catch_errors middleware the entire
    request is dumped into the log including sensitive information like
    tokens. Filter that information before outputting the failed request.

    Closes-Bug: #1628031
    Change-Id: I2563403993513c37751576223275350cac2e0937

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.middleware 3.8.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.middleware 3.19.1 release.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0001-Filter-token-data-out-of-catch_errors-middleware.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[James Page (james-page)]

Impacts 3.8.0-2 in 16.04.

[James Page (james-page)]

Reflected in https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-2592.html

[Tobias Urdin (tobias-urdin)]

Any update on releasing fix in Xenial?

[Seth Arnold (seth-arnold)]

Tobias, the Ubuntu security team has marked this with a 'Low' priority, which means we'll fix this if a 'Medium' priority (or higher) issue is found, or if several other 'Low' issues can be fixed simultaneously.

Have we miscategorized this issue?

Alternatively, we could probably sponsor an update.

Thanks

[Corey Bryant (corey.bryant)]

Attaching patch for Ubuntu xenial package.

[Seth Arnold (seth-arnold)]

Thank you Corey; I've copied a package to the Ubuntu Security Proposed PPA: https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+packages

Let us know if this package works well and we can release it Monday.

Thanks!

[Corey Bryant (corey.bryant)]

Hi Seth, this is working as expected in your PPA. Details are here: https://paste.ubuntu.com/p/Gjm9pfdXVQ/

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.middleware - 3.8.0-2ubuntu1

---------------
python-oslo.middleware (3.8.0-2ubuntu1) xenial-security; urgency=medium

  * SECURITY UPDATE: Information disclosure in log file (LP: #1628031)
    - d/p/filter-token-data-out-of-catch_errors-middleware.patch:
      ensure sensitive token data is not written to log file.
    - CVE-2017-2592

 -- Corey Bryant <email address hidden> Thu, 10 May 2018 10:00:18 -0400

[Seth Arnold (seth-arnold)]

Hi Corey, thanks for the update and testing. The USN is now live: https://usn.ubuntu.com/3666-1/

Thanks