Traceback is sent to client

[Jason Dunsmore]

Tracebacks are being sent to the client and, in turn, expose server internals that could be used in an exploit:

stack@devstack3:/opt/stack/python-heatclient$ stack_delete_last
ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 383, in handle_one_response
    try:
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 312, in __call__
    """
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/heat/heat/common/auth_password.py", line 64, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 312, in __call__
    """
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 589, in __call__
    action_result = self.dispatch(self.controller, action,
  File "/opt/stack/heat/heat/common/wsgi.py", line 634, in dispatch
    except Exception:
  File "/opt/stack/heat/heat/api/openstack/v1/util.py", line 32, in handle_stack_method
    return handle_stack_method
  File "/opt/stack/heat/heat/api/openstack/v1/util.py", line 48, in handle_stack_method
    return handle_stack_method
  File "/opt/stack/heat/heat/api/openstack/v1/stacks.py", line 303, in delete
  File "/opt/stack/heat/heat/rpc/client.py", line 153, in delete_stack
    stack_identity=stack_identity))
  File "/opt/stack/heat/heat/openstack/common/rpc/proxy.py", line 126, in call
    result = rpc.call(context, real_topic, msg, timeout)
  File "/opt/stack/heat/heat/openstack/common/rpc/__init__.py", line 140, in call
    _check_for_lock()
  File "/opt/stack/heat/heat/openstack/common/rpc/impl_kombu.py", line 799, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/opt/stack/heat/heat/openstack/common/rpc/amqp.py", line 537, in call
    rv = list(rv)
  File "/opt/stack/heat/heat/openstack/common/rpc/amqp.py", line 502, in __iter__
    raise result
ActionInProgress_Remote: Stack test1376073183 already has an action (CREATE) in progress
Traceback (most recent call last):

  File "/opt/stack/heat/heat/openstack/common/rpc/amqp.py", line 424, in _process_data
    **args)

  File "/opt/stack/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)

  File "/opt/stack/heat/heat/engine/service.py", line 55, in wrapped
    return func(self, ctx, *args, **kwargs)

  File "/opt/stack/heat/heat/engine/service.py", line 392, in delete_stack
    action=st.action)

ActionInProgress: Stack test1376073183 already has an action (CREATE) in progress

The full traceback should remain in the engine debug output only unless the engine is explicitly configured to send the info to the client.

[Steven Hardy (shardy)]

Is this still happening w/latest master? Several patches have been merged recently around exception formatting, so would be good to check if this can be closed as a dupe of one of those bugs..

[Jason Dunsmore (jasondunsmore)]

Yes, it is.

stack@devstack3:~$ stack_update_last http://dunsmor.com/heat/heat-templates/rackspace/Wordpress_Single_Instance.yaml "Foo=bar;DBUsername=wp;DBPassword=verybadpassword2"
ERROR: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 306, in __call__
    eventlet.patcher.monkey_patch(all=False, socket=True)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/heat/heat/common/auth_password.py", line 74, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 306, in __call__
    eventlet.patcher.monkey_patch(all=False, socket=True)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/heat/heat/common/wsgi.py", line 612, in __call__
    serialized by requested content type.
UnknownUserParameter_Remote: The Parameter (Foo) was not defined in template.
Traceback (most recent call last):

  File "/opt/stack/heat/heat/openstack/common/rpc/amqp.py", line 435, in _process_data
    **args)

  File "/opt/stack/heat/heat/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctx...

[Clint Byrum (clint-fewbar)]

Debugging this. It looks like eventlet.wsgi's default behavior is to show traceback to the client on any unhandled exceptions in WSGI apps.

There are two things that need to happen I think then.

1) We need to handle HeatException in the WSGI middleware so it never bubbles up to eventlet, but so that the message is displayed to the user as intended.
2) We need to apply the server's debug flag to eventlet.wsgi.Server, as passing debug=False to eventlet.wsgi.Server's constructor will suppress any unhandled tracebacks being sent to the user (they will just get a blank 500 error). Indescriminate Exception's are already logged, so deployers can still have a chance at reporting deep unhandled exceptions without exposing tracebacks to users.

Part 1 is this bug. Part 2 is bug #1222008

[Clint Byrum (clint-fewbar)]

Ok actually my api-paste.ini was out of date and did not have faultwrap in the pipeline. Adding it makes the errors appear not to have the traceback when shown in the CLI, but they are still there, just now hidden in the json response. So that still needs handling.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/45515
Committed: http://github.com/openstack/heat/commit/9e9c747b4ec0f1972da1d122e46402b00cd0911f
Submitter: Jenkins
Branch: master

commit 9e9c747b4ec0f1972da1d122e46402b00cd0911f
Author: Clint Byrum <email address hidden>
Date: Fri Sep 6 20:53:58 2013 -0700

    Only send traceback to users when in debug mode

    API services currently send the traceback to clients. While the client
    hides it from user view, it is still present in the response, exposing
    the service to details of the engine that administrators likely would
    not like to have exposed.

    Fixes bug #1210623

    Change-Id: I554ba24b7ac9166e28a8a0a10f566ed9cfa03014