Cue

Setting debug mode also causes Pecan to run in debug mode

Bug #1425206 reported by Jim Rollenhagen on 2015-02-24
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Critical
Jim Rollenhagen
Cue
Critical
Vipul Sabhaya
Gnocchi
Fix Released
Undecided
Chris Dent
Ironic
Critical
Unassigned
OpenStack Security Advisory
Undecided
Unassigned
OpenStack Security Notes
Undecided
Robert Clark

Bug Description

When debug mode is set (via --debug or CONF.debug=True), pecan is also put into debug mode. In debug mode, pecan serves HTML for every 500 that:

1) gives the full traceback
2) gives the full list of environment variables
3) allows the user to retry the request with a breakpoint inserted, which locks up the service in a pdb shell

This means that running the API service in debug mode can result in both system information leaks (which could contain secrets) and a denial of service.

Ironic is not the only service affected in this way; ceilometer's pecan_debug option defaults to CONF.debug, and the following projects also use CONF.debug to set pecan's debug mode:
openstack/tuskar
openstack/kite
openstack-infra/storyboard
stackforge/libra
stackforge/blazar
stackforge/cue

This is not an exhaustive list, but I believe I have checked all openstack/ projects and did a fairly thorough search on Github. Ryan Petrello and Doug Hellman helped to make this list.

Changed in ironic:
status: New → Confirmed

DEBUG mode related leak have not been considered as a vulnerability and didn't warrant an advisory. Moreover Ironic nor the other mentioned project are part of the security supported project.

Feel free to remove the private security settings if it makes sense.

Changed in ossa:
status: New → Incomplete
status: Incomplete → Won't Fix

FWIW, ceilometer is a security supported project.

I think the fact that this many projects (and possibly more) are doing a thing that can be *really* bad is worth at least an advisory, so deployers are aware. It's not uncommon (nor a bad idea) to run OpenStack services in debug mode, and many of the largest clouds do exactly this.

Oups I misread, adding ceilometer task.

@Jim well I agree with the need to fix this, especially if it's not an intended behavior. However, to my knowledge, it seems pretty clear that running OpenStack in DEBUG mode is not something you want in production or in any secure deployment. I'm not sure we would consider this OSSA materials since (1) I don't think we should backport it and alter behavior for already-released versions and (2) we always considered leaks in DEBUG logs as bugs rather than vulnerabilities.

Though you mention the ability to add breakpoints, which could be used maliciously... but preventing the use of a debugger wouldn't defeat the purpose of a DEBUG mode ?

The reality is that deployers *do* run OpenStack in debug mode, for better or worse. It's often necessary to be able to track down production issues. Of course, if it was possible to get DEBUG level logs without running in DEBUG mode, that would probably solve that problem.

I'm not suggesting that we should never allow Pecan's debug mode, but that it should be a separate configuration from the global debug option.

If the OSSA doesn't want to handle this at all, then by all means open this up. However, I tend to think that's ignoring how people actually deploy OpenStack in reality.

gordon chung (chungg) on 2015-02-24
Changed in ceilometer:
status: New → Triaged
importance: Undecided → Critical
milestone: none → kilo-3
Changed in ossa:
status: Won't Fix → Incomplete

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I'm still not convinced that debug related issues should be treated as vulnerabilities and/or under embargo, though I put back the OSSA task as incomplete for further discussion...

Patch for Ironic master included, unit/pep8 tests pass.

Including patch for Ceilometer master branch; I have not successfully run tests on it yet but Gordon has volunteered to.

Changed in ironic:
importance: Undecided → Critical
status: Confirmed → Triaged
milestone: none → kilo-3
gordon chung (chungg) wrote :

i ran Jim's patch locally and it passed tox -epy27.

gordon chung (chungg) wrote :

re: ceilometer patch. i'm ok with it -- seems like a simple enough fix. interested in comments from other cores.

I've run Jim's patch locally on top of Ironic master and confirmed that it fixes the described problem.

Specifically, running the API service in debug mode no longer causes pecan to also be in debug mode, and an operator must explicitly set pecan_debug=True if they want to enable HTTP clients to set breakpoints or see server ENV vars.

I've just tested Ironic 2014.2.1 (stable/juno) and confirmed it is also affected by this, and confirmed that Jim's patch fixes the issue there as well. However, there is a one-line patch conflict when applying it to stable/juno.

Thanks for attaching patches. So ceilometer already have that extra option, it's just that the default value is insecure. Beside, https://github.com/openstack/ceilometer/blob/master/ceilometer/api/app.py#L83-L88 suggest that only single worker ceilometer would be affected.

Thus I propose a class B type of bug ( https://wiki.openstack.org/wiki/Vulnerability_Management#Incident_report_taxonomy )

Tristan, what are the next steps here?

Adding ossg-coresec to confirm the proposed class B.

@Jim If ossg agree, we open the bug, push fix for master with the new default for pecan_debug and issue an OSSN for stable releases.

Eoghan Glynn (eglynn) wrote :

+1 on the proposed ceilometer patch

Robert Clark (robert-clark) wrote :

Yeah this is pretty horrible, my understanding is that most people have to run at least some services in Debug to be able to realistically maintain them.

An OSSN would be appropriate, I'll add this bug to the OSSN queue once it's become public.

Nathan Kinder (nkinder) wrote :

I agree this is a pretty bad one since it serves up all of this potentially sensitive information to end users. +1 on an OSSN once this is public.

Thanks Robert and Nathan for the quick feedback.

@Jim, feel free to open this bug now.

Changed in ossa:
status: Incomplete → Won't Fix

Patches are pushed, will open this bug in a moment once they get a +A.

Robert, go ahead and add this to OSSN queue.

Opening this bug since patch are now public.

information type: Private Security → Public Security
Changed in ceilometer:
assignee: nobody → Jim Rollenhagen (jim-rollenhagen)
status: Triaged → In Progress

Reviewed: https://review.openstack.org/159589
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=7f2a969707b706b6877d4e278c6c1c24c035b527
Submitter: Jenkins
Branch: master

commit 7f2a969707b706b6877d4e278c6c1c24c035b527
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:44:00 2015 +0000

    Create new config for pecan debug mode

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. However, it may be useful
    to use Pecan's debug mode in development, so create a config option
    to enable it, rather than disable it altogether.

    Change-Id: I5bc76b4101c563cdc168d2e55db060c1bdd0b5fe
    Closes-Bug: #1425206

Changed in ironic:
status: Triaged → Fix Committed

Reviewed: https://review.openstack.org/159591
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b2a21fba08df7da1f7c82fd41e54797e38f2fd53
Submitter: Jenkins
Branch: master

commit b2a21fba08df7da1f7c82fd41e54797e38f2fd53
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:56:53 2015 +0000

    Do not default pecan_debug to CONF.debug

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. Defaulting pecan_debug to
    CONF.debug makes this easy to accidentally do. So, default it to False
    rather than riding on top of CONF.debug.

    Change-Id: I70f9c9807d16aa50df4d5e16ba2a29575f8b165e
    Closes-Bug: #1425206
    DocImpact

Changed in ceilometer:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/159588
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=0f4d454bf2093d3d62460f88aa9288bee3286b38
Submitter: Jenkins
Branch: stable/juno

commit 0f4d454bf2093d3d62460f88aa9288bee3286b38
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:44:00 2015 +0000

    Create new config for pecan debug mode

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. However, it may be useful
    to use Pecan's debug mode in development, so create a config option
    to enable it, rather than disable it altogether.

    Change-Id: I5bc76b4101c563cdc168d2e55db060c1bdd0b5fe
    Closes-Bug: #1425206

tags: added: in-stable-juno

Fix proposed to branch: master
Review: https://review.openstack.org/162698

Changed in gnocchi:
assignee: nobody → gordon chung (chungg)
status: New → In Progress
Changed in gnocchi:
assignee: gordon chung (chungg) → Chris Dent (chdent)

Reviewed: https://review.openstack.org/162698
Committed: https://git.openstack.org/cgit/stackforge/gnocchi/commit/?id=6223aa1fd85bebda70a50af76c55b9e695f48377
Submitter: Jenkins
Branch: master

commit 6223aa1fd85bebda70a50af76c55b9e695f48377
Author: gordon chung <email address hidden>
Date: Mon Mar 9 13:24:29 2015 -0400

    default pecan debug to false

    pecan outputs extremely verbose details when debug is set. right
    or wrong, operators tend to use debug in production. this patch
    minimise security impact of arguably bad deployment choices.

    Change-Id: I69cf0befdf4acb5ddeac01cfda640ee6f86c4f38
    Closes-Bug: #1425206

Changed in gnocchi:
status: In Progress → Fix Committed
Vipul Sabhaya (vipuls) on 2015-03-16
Changed in cue:
importance: Undecided → Critical
Vipul Sabhaya (vipuls) on 2015-03-16
Changed in cue:
assignee: nobody → Vipul Sabhaya (vipuls)
Thierry Carrez (ttx) on 2015-03-19
Changed in ceilometer:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2015-03-19
Changed in ironic:
status: Fix Committed → Fix Released
Julien Danjou (jdanjou) on 2015-03-20
Changed in gnocchi:
status: Fix Committed → Fix Released
milestone: none → 1.0.0a1
Robert Clark (robert-clark) wrote :

Writing up the OSSN for this now, I presume other projects than listed above are unaffected?

Thierry Carrez (ttx) on 2015-04-30
Changed in ceilometer:
milestone: kilo-3 → 2015.1.0
Thierry Carrez (ttx) on 2015-04-30
Changed in ironic:
milestone: kilo-3 → 2015.1.0
Nathan Kinder (nkinder) wrote :

This has been published as OSSN-0046:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0046

Changed in ossn:
assignee: nobody → Robert Clark (robert-clark)
status: New → Fix Released
Vipul Sabhaya (vipuls) wrote :
Changed in cue:
status: New → Fix Committed
Vipul Sabhaya (vipuls) on 2015-11-11
Changed in cue:
milestone: none → 1.0.0
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers