Setting debug mode also causes Pecan to run in debug mode

[Jim Rollenhagen]

When debug mode is set (via --debug or CONF.debug=True), pecan is also put into debug mode. In debug mode, pecan serves HTML for every 500 that:

1) gives the full traceback
2) gives the full list of environment variables
3) allows the user to retry the request with a breakpoint inserted, which locks up the service in a pdb shell

This means that running the API service in debug mode can result in both system information leaks (which could contain secrets) and a denial of service.

Ironic is not the only service affected in this way; ceilometer's pecan_debug option defaults to CONF.debug, and the following projects also use CONF.debug to set pecan's debug mode:
openstack/tuskar
openstack/kite
openstack-infra/storyboard
stackforge/libra
stackforge/blazar
stackforge/cue

This is not an exhaustive list, but I believe I have checked all openstack/ projects and did a fairly thorough search on Github. Ryan Petrello and Doug Hellman helped to make this list.

[Tristan Cacqueray (tristan-cacqueray)]

DEBUG mode related leak have not been considered as a vulnerability and didn't warrant an advisory. Moreover Ironic nor the other mentioned project are part of the security supported project.

Feel free to remove the private security settings if it makes sense.

[Jim Rollenhagen (jim-rollenhagen)]

FWIW, ceilometer is a security supported project.

I think the fact that this many projects (and possibly more) are doing a thing that can be *really* bad is worth at least an advisory, so deployers are aware. It's not uncommon (nor a bad idea) to run OpenStack services in debug mode, and many of the largest clouds do exactly this.

[Tristan Cacqueray (tristan-cacqueray)]

Oups I misread, adding ceilometer task.

@Jim well I agree with the need to fix this, especially if it's not an intended behavior. However, to my knowledge, it seems pretty clear that running OpenStack in DEBUG mode is not something you want in production or in any secure deployment. I'm not sure we would consider this OSSA materials since (1) I don't think we should backport it and alter behavior for already-released versions and (2) we always considered leaks in DEBUG logs as bugs rather than vulnerabilities.

Though you mention the ability to add breakpoints, which could be used maliciously... but preventing the use of a debugger wouldn't defeat the purpose of a DEBUG mode ?

[Jim Rollenhagen (jim-rollenhagen)]

The reality is that deployers *do* run OpenStack in debug mode, for better or worse. It's often necessary to be able to track down production issues. Of course, if it was possible to get DEBUG level logs without running in DEBUG mode, that would probably solve that problem.

I'm not suggesting that we should never allow Pecan's debug mode, but that it should be a separate configuration from the global debug option.

If the OSSA doesn't want to handle this at all, then by all means open this up. However, I tend to think that's ignoring how people actually deploy OpenStack in reality.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I'm still not convinced that debug related issues should be treated as vulnerabilities and/or under embargo, though I put back the OSSA task as incomplete for further discussion...

[Jim Rollenhagen (jim-rollenhagen)]

Patch for Ironic master included, unit/pep8 tests pass.

[Jim Rollenhagen (jim-rollenhagen)]

Including patch for Ceilometer master branch; I have not successfully run tests on it yet but Gordon has volunteered to.

[gordon chung (chungg)]

i ran Jim's patch locally and it passed tox -epy27.

[gordon chung (chungg)]

re: ceilometer patch. i'm ok with it -- seems like a simple enough fix. interested in comments from other cores.

[aeva black (tenbrae)]

I've run Jim's patch locally on top of Ironic master and confirmed that it fixes the described problem.

Specifically, running the API service in debug mode no longer causes pecan to also be in debug mode, and an operator must explicitly set pecan_debug=True if they want to enable HTTP clients to set breakpoints or see server ENV vars.

[aeva black (tenbrae)]

I've just tested Ironic 2014.2.1 (stable/juno) and confirmed it is also affected by this, and confirmed that Jim's patch fixes the issue there as well. However, there is a one-line patch conflict when applying it to stable/juno.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for attaching patches. So ceilometer already have that extra option, it's just that the default value is insecure. Beside, https://github.com/openstack/ceilometer/blob/master/ceilometer/api/app.py#L83-L88 suggest that only single worker ceilometer would be affected.

Thus I propose a class B type of bug ( https://wiki.openstack.org/wiki/Vulnerability_Management#Incident_report_taxonomy )

[Jim Rollenhagen (jim-rollenhagen)]

Tristan, what are the next steps here?

[Tristan Cacqueray (tristan-cacqueray)]

Adding ossg-coresec to confirm the proposed class B.

@Jim If ossg agree, we open the bug, push fix for master with the new default for pecan_debug and issue an OSSN for stable releases.

[Eoghan Glynn (eglynn)]

+1 on the proposed ceilometer patch

[Robert Clark (robert-clark)]

Yeah this is pretty horrible, my understanding is that most people have to run at least some services in Debug to be able to realistically maintain them.

An OSSN would be appropriate, I'll add this bug to the OSSN queue once it's become public.

[Nathan Kinder (nkinder)]

I agree this is a pretty bad one since it serves up all of this potentially sensitive information to end users. +1 on an OSSN once this is public.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks Robert and Nathan for the quick feedback.

@Jim, feel free to open this bug now.

[Jim Rollenhagen (jim-rollenhagen)]

Patches are pushed, will open this bug in a moment once they get a +A.

Robert, go ahead and add this to OSSN queue.

[Tristan Cacqueray (tristan-cacqueray)]

Opening this bug since patch are now public.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159589
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=7f2a969707b706b6877d4e278c6c1c24c035b527
Submitter: Jenkins
Branch: master

commit 7f2a969707b706b6877d4e278c6c1c24c035b527
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:44:00 2015 +0000

    Create new config for pecan debug mode

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. However, it may be useful
    to use Pecan's debug mode in development, so create a config option
    to enable it, rather than disable it altogether.

    Change-Id: I5bc76b4101c563cdc168d2e55db060c1bdd0b5fe
    Closes-Bug: #1425206

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159591
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b2a21fba08df7da1f7c82fd41e54797e38f2fd53
Submitter: Jenkins
Branch: master

commit b2a21fba08df7da1f7c82fd41e54797e38f2fd53
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:56:53 2015 +0000

    Do not default pecan_debug to CONF.debug

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. Defaulting pecan_debug to
    CONF.debug makes this easy to accidentally do. So, default it to False
    rather than riding on top of CONF.debug.

    Change-Id: I70f9c9807d16aa50df4d5e16ba2a29575f8b165e
    Closes-Bug: #1425206
    DocImpact

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159588
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=0f4d454bf2093d3d62460f88aa9288bee3286b38
Submitter: Jenkins
Branch: stable/juno

commit 0f4d454bf2093d3d62460f88aa9288bee3286b38
Author: Jim Rollenhagen <email address hidden>
Date: Tue Feb 24 21:44:00 2015 +0000

    Create new config for pecan debug mode

    Pecan's debug mode can be terribly insecure; 500 errors return a
    Python traceback, the full list of environment variables, and a
    button to replay the request with a breakpoint.

    Deployers often run OpenStack services in debug mode; doing so should
    not open the service up to these flaws. However, it may be useful
    to use Pecan's debug mode in development, so create a config option
    to enable it, rather than disable it altogether.

    Change-Id: I5bc76b4101c563cdc168d2e55db060c1bdd0b5fe
    Closes-Bug: #1425206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/162698

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/162698
Committed: https://git.openstack.org/cgit/stackforge/gnocchi/commit/?id=6223aa1fd85bebda70a50af76c55b9e695f48377
Submitter: Jenkins
Branch: master

commit 6223aa1fd85bebda70a50af76c55b9e695f48377
Author: gordon chung <email address hidden>
Date: Mon Mar 9 13:24:29 2015 -0400

    default pecan debug to false

    pecan outputs extremely verbose details when debug is set. right
    or wrong, operators tend to use debug in production. this patch
    minimise security impact of arguably bad deployment choices.

    Change-Id: I69cf0befdf4acb5ddeac01cfda640ee6f86c4f38
    Closes-Bug: #1425206

[Robert Clark (robert-clark)]

Writing up the OSSN for this now, I presume other projects than listed above are unaffected?

[Robert Clark (robert-clark)]

OSSN Draft https://review.openstack.org/#/c/169388/

[Nathan Kinder (nkinder)]

This has been published as OSSN-0046:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0046

[Vipul Sabhaya (vipuls)]

Cue fixed https://review.openstack.org/#/c/159614/3