Cinderclient stack traces when BadRequest (400) is returned

Bug #1158505 reported by Sean Dague
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Sean Dague

Bug Description

In a successful run of tempest, we have negative tests for invalid volume sizes. When we set that we get a stack trace from cinderclient as follows:

<2013-03-19 18:35:45.729 TRACE nova.api.openstack>
Traceback (most recent call last): File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 81, in __call__
    return req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/opt/stack/new/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 450, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 890, in __call__
    content_type, body, accept)
  File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 942, in _process_stack
    action_result = self.dispatch(meth, request, action_args)
  File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 1022, in dispatch
    return method(req=request, **action_args)
  File "/opt/stack/new/nova/nova/api/openstack/compute/contrib/volumes.py", line 254, in create
    availability_zone=availability_zone
  File "/opt/stack/new/nova/nova/volume/cinder.py", line 268, in create
    item = cinderclient(context).volumes.create(size, **kwargs)
  File "/opt/stack/new/python-cinderclient/cinderclient/v1/volumes.py", line 171, in create
    return self._create('/volumes', body, 'volume')
  File "/opt/stack/new/python-cinderclient/cinderclient/base.py", line 149, in _create
    resp, body = self.api.client.post(url, body=body)
  File "/opt/stack/new/python-cinderclient/cinderclient/client.py", line 188, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/opt/stack/new/python-cinderclient/cinderclient/client.py", line 153, in _cs_request
    **kwargs)
  File "/opt/stack/new/python-cinderclient/cinderclient/client.py", line 136, in request
    raise exceptions.from_response(resp, body)
BadRequest: Invalid input received: Volume size '0' must be an integer and greater than 0 (HTTP 400) (Request-ID: req-ba526415-c315-4151-a869-ee68a9019289)

We should catch this exception low enough that we don't stack trace on it.

Tags: stacktrace
Sean Dague (sdague)
Changed in nova:
importance: Undecided → High
Sean Dague (sdague)
tags: added: stacktrace
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/25072

Changed in nova:
assignee: nobody → Sean Dague (sdague)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/25072
Committed: http://github.com/openstack/nova/commit/3801a4d2f4c59dbfda49131ddde22fcb3976d651
Submitter: Jenkins
Branch: master

commit 3801a4d2f4c59dbfda49131ddde22fcb3976d651
Author: Sean Dague <email address hidden>
Date: Thu Mar 21 18:48:30 2013 -0400

    translate cinder BadRequest exception

    if we attempt to create a volume with values that are invalid (like
    a non numeric value for size) we properly get a BadRequest returned
    from cinder. However we inproperly pass that cinderclient exception
    all the way to the top of the request process, causing a stack trace
    to appear in the logs because this is an exception type Nova doesn't
    understand.

    This situation is triggered by Tempest on every run, and while the
    return is successful, the stack trace might spook people.

    Fix this by increasing the scope of the existing
    _reraise_translated_volume_exception() method. Longer term it would
    make sense to have a decorator that handles all possible exceptions
    we'd expect from cinder client and turn them into Nova exceptions.

    This reduces the number of stack traces in nova-api by 6 on a
    successful Tempest run.

    Fixes bug #1158505

    Step towards blueprint no-stacktraces-in-logs

    Change-Id: Ifcfbd5eb11fe9f038f648ca5291499290b7126b1

Changed in nova:
status: In Progress → Fix Committed
Sean Dague (sdague)
tags: added: grizzly-backport-potential
Changed in nova:
milestone: none → grizzly-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25444

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (milestone-proposed)

Reviewed: https://review.openstack.org/25444
Committed: http://github.com/openstack/nova/commit/9bfad01d9082992d10ece85d141c5c5760f5b935
Submitter: Jenkins
Branch: milestone-proposed

commit 9bfad01d9082992d10ece85d141c5c5760f5b935
Author: Sean Dague <email address hidden>
Date: Thu Mar 21 18:48:30 2013 -0400

    translate cinder BadRequest exception

    if we attempt to create a volume with values that are invalid (like
    a non numeric value for size) we properly get a BadRequest returned
    from cinder. However we inproperly pass that cinderclient exception
    all the way to the top of the request process, causing a stack trace
    to appear in the logs because this is an exception type Nova doesn't
    understand.

    This situation is triggered by Tempest on every run, and while the
    return is successful, the stack trace might spook people.

    Fix this by increasing the scope of the existing
    _reraise_translated_volume_exception() method. Longer term it would
    make sense to have a decorator that handles all possible exceptions
    we'd expect from cinder client and turn them into Nova exceptions.

    This reduces the number of stack traces in nova-api by 6 on a
    successful Tempest run.

    Fixes bug #1158505

    Step towards blueprint no-stacktraces-in-logs

    Change-Id: Ifcfbd5eb11fe9f038f648ca5291499290b7126b1
    (cherry picked from commit 3801a4d2f4c59dbfda49131ddde22fcb3976d651)

Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc2 → 2013.1
Alan Pevec (apevec)
tags: removed: grizzly-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.