Cinderclient Doesn't Return A Useful Error When Trying To Create A Volume Larger Than The Quota Allocation

Bug #1013417 reported by John Postlethwait
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Mike Perez
Icehouse
Fix Released
Undecided
Unassigned
python-cinderclient
Fix Released
Undecided
Mike Perez

Bug Description

Actually, it is nearly useless. It just returns an exception that it casts from a HTTP 500.

My quota limit is 1000GB, here I try to make a volume that is 2000GB

g = cinderclient(request).volumes.create(size, display_name=name, display_description=description)

cinderclient connection created using token "e3fbb3c2d94949b0975db11de85bebc5" and url "http://10.145.1.51:8776/v1/9da18fcaedf74eb7b1cf73b67b5b870c"

REQ: curl -i http://10.145.1.51:8776/v1/9da18fcaedf74eb7b1cf73b67b5b870c/volumes -X POST -H "X-Auth-Project-Id: 9da18fcaedf74eb7b1cf73b67b5b870c" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: e3fbb3c2d94949b0975db11de85bebc5"

REQ BODY: {"volume": {"snapshot_id": null, "display_name": "My Vol", "volume_type": null, "display_description": "", "size": 2000}}

RESP:{'date': 'Thu, 14 Jun 2012 22:14:02 GMT', 'status': '500', 'content-length': '128', 'content-type': 'application/json; charset=UTF-8', 'x-compute-request-id': 'req-316c81e2-3407-4df0-8b0e-190bf63f549b'} {"computeFault": {"message": "The server has either erred or is incapable of performing the requested operation.", "code": 500}}

*** ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-316c81e2-3407-4df0-8b0e-190bf63f549b)

This is basically useless from an end-user perspective and doesn't allow us to tell users of Horizon anything useful about why this error'd. :( It should probably be a 406, not a 500, and the error message should be "Cannot create a volume of 2000GB because your quota is currently 1000GB." Or something along those lines...

description: updated
description: updated
no longer affects: horizon
Thierry Carrez (ttx)
tags: added: python-cinderclient
Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
tags: added: volume
Thierry Carrez (ttx)
affects: cinder → python-cinderclient
Mike Perez (thingee)
Changed in python-cinderclient:
status: New → Confirmed
Mike Perez (thingee)
Changed in python-cinderclient:
assignee: nobody → Mike Perez (thingee)
Revision history for this message
Mike Perez (thingee) wrote :

Talked with John about this. This is going to involve quite a bit of change. Need to communicate this more with people before I proceed.

https://blueprints.launchpad.net/cinder/+spec/clearer-api-error-responses

Revision history for this message
Mike Perez (thingee) wrote :

If the request gets any sort of exception, it appears we're forcing a 500 with no explanation. This was done because sensitive information was being passed to the client. Might be time to look into a way of passing back useful information while avoiding this issue:

https://bugs.launchpad.net/nova/+bug/868360
https://bugs.launchpad.net/nova/+bug/874472

The commit in question: http://github.com/openstack/nova/commit/2431b7848d633dc67ad684b4d1cc79468df24568

Mike Perez (thingee)
Changed in python-cinderclient:
status: Confirmed → Fix Committed
Changed in nova:
assignee: nobody → Sergio Cazzolato (sergio-j-cazzolato)
assignee: Sergio Cazzolato (sergio-j-cazzolato) → nobody
Revision history for this message
Mike Perez (thingee) wrote :
Download full text (3.8 KiB)

On the nova side, this just says invalid block device, which is caught by a catch all exception.

2014-02-04 20:45:32.689 43992 ERROR nova.compute.manager [req-10a4ded5-dbe5-4951-a787-0face80a95f5 90deab5f729c4176addf3f063d0d2a87 cb99317047bb4935b43f56fa7f48acdf] [instance: 0bd1614d-da1a-4e85-901a-08753ce
97954] Instance failed block device setup
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] Traceback (most recent call last):
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 1386, in _prep_block_devi
ce
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] self._await_block_device_map_created))
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/nova/virt/block_device.py", line 283, in attach_block_de
vices
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] block_device_mapping)
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/nova/virt/block_device.py", line 236, in attach
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] '', '', image_id=self.image_id)
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/nova/volume/cinder.py", line 306, in create
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] item = cinderclient(context).volumes.create(size, **kwargs)
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/cinderclient/v1/volumes.py", line 187, in create
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] return self._create('/volumes', body, 'volume')
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/cinderclient/base.py", line 153, in _create
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] resp, body = self.api.client.post(url, body=body)
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/cinderclient/client.py", line 210, in post
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] return self._cs_request(url, 'POST', **kwargs)
2014-02-04 20:45:32.689 43992 TRACE nova.compute.manager [instance: 0bd1614d-da1a-4e85-901a-08753ce97954] File "/opt/nova/lib/python2.7/site-packages/cinderclient/client.py", line 174, in _cs_request
2014-02-04 20:45:32.689 4...

Read more...

Changed in nova:
assignee: nobody → Mike Perez (thingee)
status: Confirmed → In Progress
Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/71147
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4711ff3adff2b4c9a54f5fb4448ace6558f82f6b
Submitter: Jenkins
Branch: master

commit 4711ff3adff2b4c9a54f5fb4448ace6558f82f6b
Author: Mike Perez <email address hidden>
Date: Tue Feb 4 15:00:10 2014 -0800

    Raise descriptive error for over volume quota

    If you attempt to create a new instance booting from an image and you
    were over your quota on cinder, the nova manager would see that as an
    invalid block device. This raises a more descriptive error message in
    the logs.

    Closes-bug: #1013417
    Change-Id: Ie50b97c31580da7fc38c6891d1f3646b91f8aa10

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/120172

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/120172
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5ec3cd30242350c1d0e759d8000e778c8cc1b48b
Submitter: Jenkins
Branch: stable/icehouse

commit 5ec3cd30242350c1d0e759d8000e778c8cc1b48b
Author: Mike Perez <email address hidden>
Date: Tue Feb 4 15:00:10 2014 -0800

    Raise descriptive error for over volume quota

    If you attempt to create a new instance booting from an image and you
    were over your quota on cinder, the nova manager would see that as an
    invalid block device. This raises a more descriptive error message in
    the logs.

    Closes-bug: #1013417
    Change-Id: Ie50b97c31580da7fc38c6891d1f3646b91f8aa10
    (cherry picked from commit 4711ff3adff2b4c9a54f5fb4448ace6558f82f6b)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-1 → 2014.2
Changed in python-cinderclient:
milestone: none → 1.3.1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.