[nova][volumes] Exceeding volumes, gigabytes and floating_ips quotas returns general uninformative HTTP 500 error

Bug #1021373 reported by David Naori on 2012-07-05
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Eoghan Glynn
Essex
Medium
Eoghan Glynn
nova (Ubuntu)
Undecided
Unassigned
Precise
Undecided
Unassigned

Bug Description

Description of problem:
Exceeding volumes, gigabytes and floating_ips quotas returns general uninformative HTTP 500 error

#nova volume-create 1
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500)

2012-07-03 18:36:59 AUDIT nova.api.openstack.volume.volumes [req-35f94ead-5c07-4bb5-a62f-b2d26041e1b7 29af43364ced4611bf6e093a04d4c144 3f1e394b238d4f26b742e9685a9a1a57] Create volume of 1 GB
2012-07-03 18:36:59 WARNING nova.volume.api [req-35f94ead-5c07-4bb5-a62f-b2d26041e1b7 29af43364ced4611bf6e093a04d4c144 3f1e394b238d4f26b742e9685a9a1a57] Quota exceeded for 3f1e394b238d4f26b742e9685a9a1a57, tried to create 1G volume
2012-07-03 18:36:59 ERROR nova.api.openstack [req-35f94ead-5c07-4bb5-a62f-b2d26041e1b7 29af43364ced4611bf6e093a04d4c144 3f1e394b238d4f26b742e9685a9a1a57] Caught error: Quota exceeded: code=VolumeSizeTooLarge
2012-07-03 18:36:59 TRACE nova.api.openstack Traceback (most recent call last):
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 41, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return req.get_response(self.application)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response
2012-07-03 18:36:59 TRACE nova.api.openstack application, catch_exc_info=False)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application
2012-07-03 18:36:59 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 176, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return self.app(env, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack return resp(environ, start_response)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 147, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 208, in call_func
2012-07-03 18:36:59 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 803, in __call__
2012-07-03 18:36:59 TRACE nova.api.openstack content_type, body, accept)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 851, in _process_stack
2012-07-03 18:36:59 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 926, in dispatch
2012-07-03 18:36:59 TRACE nova.api.openstack return method(req=request, **action_args)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/api/openstack/volume/volumes.py", line 249, in create
2012-07-03 18:36:59 TRACE nova.api.openstack **kwargs)
2012-07-03 18:36:59 TRACE nova.api.openstack File "/usr/lib/python2.6/site-packages/nova/volume/api.py", line 87, in create
2012-07-03 18:36:59 TRACE nova.api.openstack raise exception.QuotaError(code="VolumeSizeTooLarge")
2012-07-03 18:36:59 TRACE nova.api.openstack QuotaError: Quota exceeded: code=VolumeSizeTooLarge
2012-07-03 18:36:59 TRACE nova.api.openstack
2012-07-03 18:36:59 INFO nova.api.openstack [req-35f94ead-5c07-4bb5-a62f-b2d26041e1b7 29af43364ced4611bf6e093a04d4c144 3f1e394b238d4f26b742e9685a9a1a57] http://camel-nova:8776/v1/3f1e394b238d4f26b742e9685a9a1a57/volumes returned with HTTP 500

How reproducible:
100%

Related branches

Eoghan Glynn (eglynn) on 2012-07-05
Changed in nova:
status: New → Confirmed
assignee: nobody → Eoghan Glynn (eglynn)
Eoghan Glynn (eglynn) wrote :

Apart from obscuring the true source of the error, this generic-500 approach in the native openstack API is also inconsistent with the EC2 API, where the exception type and message are exposed in the /Response/Errors/Error/Code and /Response/Errors/Error/Message elements respectively.

Vincent Untz (vuntz) wrote :

I'm explicitly not submitting to gerrit, since a proper patch would cover all the other places where we would need a similar change. I can submit if wanted, though.

Eoghan Glynn (eglynn) on 2012-07-06
Changed in nova:
status: Confirmed → In Progress
Eoghan Glynn (eglynn) wrote :

Note that I've kept the 500 status, but with a more informative message, as opposed to switching to for example a 413 status - the reasoning here is that the Retry-After logic is not useful in a quota breach case, as opposed to a rate-limit throttling.

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/9446

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/9534

Reviewed: https://review.openstack.org/9429
Committed: http://github.com/openstack/nova/commit/e9d21589d39355ffc126e360cc2ba7311e014edb
Submitter: Jenkins
Branch: master

commit e9d21589d39355ffc126e360cc2ba7311e014edb
Author: Eoghan Glynn <email address hidden>
Date: Fri Jul 6 10:10:28 2012 +0000

    Expose over-quota exceptions via native API.

    Fixes bug LP 1021373.

    Previously an over-quota condition would be exposed via the EC2 API,
    but hidden in the corresponding call via the native API (in the sense
    of the exception detail being replaced with a generic 500 Server Error
    response).

    We now report any NovaException declared to be safe. In this patch,
    the set of safe exception types includes any subclass of QuotaError,
    but in subsequent patches the net should be widened to include
    exceptions that do not expose senstive information.

    Change-Id: I3cc36337c7e67cf487ca49de646c437c217ae538

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/9445
Committed: http://github.com/openstack/nova/commit/a3576bbb0c7090b97b0b02c88ffa81915db6290b
Submitter: Jenkins
Branch: master

commit a3576bbb0c7090b97b0b02c88ffa81915db6290b
Author: Eoghan Glynn <email address hidden>
Date: Tue Jul 10 08:16:27 2012 +0100

    Return 413 status on over-quota in the native API.

    Related to LP 1021373.

    Previously we returned a generic 500 Server Error on an over-quota
    conditions, whereas this is arguably more appropriately reported
    as a 413 status.

    Change-Id: I5c1cdc9db54804c512d60e4179c1faa13516d6f9

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/9430
Committed: http://github.com/openstack/nova/commit/4c49df742d1730b124f81e30f12e61602966a819
Submitter: Jenkins
Branch: stable/essex

commit 4c49df742d1730b124f81e30f12e61602966a819
Author: Eoghan Glynn <email address hidden>
Date: Fri Jul 6 10:10:28 2012 +0000

    Expose over-quota exceptions via native API.

    Fixes bug LP 1021373.

    Previously an over-quota condition would be exposed via the EC2 API,
    but hidden in the corresponding call via the native API (in the sense
    of the exception detail being replaced with a generic 500 Server Error
    response).

    We now report any NovaException declared to be safe. In this patch,
    the set of safe exception types includes any subclass of QuotaError,
    but in subsequent patches the net should be widened to include
    exceptions that do not expose senstive information.

    Change-Id: I3cc36337c7e67cf487ca49de646c437c217ae538

Reviewed: https://review.openstack.org/9446
Committed: http://github.com/openstack/nova/commit/6e873bccc078e1013987698cd71c69deebf2e99f
Submitter: Jenkins
Branch: stable/essex

commit 6e873bccc078e1013987698cd71c69deebf2e99f
Author: Eoghan Glynn <email address hidden>
Date: Mon Jul 9 21:09:06 2012 +0100

    Return 413 status on over-quota in the native API.

    Related to LP 1021373.

    Previously, we returned a generic 500 Server Error on an over-quota
    conditions, whereas this is arguably more appropriately reported
    as a 413 status.

    Change-Id: I5c1cdc9db54804c512d60e4179c1faa13516d6f9

Reviewed: https://review.openstack.org/9534
Committed: http://github.com/openstack/nova/commit/4b89b4fd3665a6eaaa63911102070fd93f20f8bc
Submitter: Jenkins
Branch: stable/essex

commit 4b89b4fd3665a6eaaa63911102070fd93f20f8bc
Author: Eoghan Glynn <email address hidden>
Date: Tue Jul 10 12:11:28 2012 +0100

    Handle local & remote exceptions consistently.

    Related to LP 1021373.

    When the floating_ips quota threshold is exceeded, this occurs remotely
    in the networking service and hence is reported back to the API service
    over RPC, via a RemoteError wrapping the original QuotaError exception.

    Previously the handling of such RemoteErrors in the native API was
    inconsistent with exceptions raised locally within the API service.

    Now in both cases, we expose the exception explantion, status etc. in the
    response, if the exception type is declared as safe.

    This fix is not required on master, as the Folsom RPC failure deserialization
    logic renders it unnecessary.

    Change-Id: Icf7dcafd1f942f5d81d12587775d6ade4a176932

Eoghan Glynn (eglynn) on 2012-08-14
Changed in nova:
importance: Undecided → Medium
milestone: none → folsom-3
Thierry Carrez (ttx) on 2012-08-16
Changed in nova:
status: Fix Committed → Fix Released
Dave Walker (davewalker) on 2012-08-24
Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Precise):
status: New → Confirmed

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/9445
Stable review: https://review.openstack.org/9534

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Adam Gandelman (gandelman-a) wrote :

Test coverage log.

tags: added: verification-done
Adam Gandelman (gandelman-a) wrote :

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/9445
Stable review: https://review.openstack.org/9534

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Adam Gandelman (gandelman-a) wrote :

Test coverage log.

Launchpad Janitor (janitor) wrote :
Download full text (5.4 KiB)

This bug was fixed in the package nova - 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1

---------------
nova (2012.1.3+stable-20120827-4d2a4afe-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot, fixes FTBFS in -proposed. (LP: #1041120)
  * Resynchronize with stable/essex (4d2a4afe):
    - [5d63601] Inappropriate exception handling on kvm live/block migration
      (LP: #917615)
    - [ae280ca] Deleted floating ips can cause instance delete to fail
      (LP: #1038266)

nova (2012.1.3+stable-20120824-86fb7362-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Dropped, superseded by new snapshot:
    - debian/patches/CVE-2012-3447.patch: [d9577ce]
    - debian/patches/CVE-2012-3371.patch: [25f5bd3]
    - debian/patches/CVE-2012-3360+3361.patch: [b0feaff]
  * Resynchronize with stable/essex (86fb7362):
    - [86fb736] Libvirt driver reports incorrect error when volume-detach fails
      (LP: #1029463)
    - [272b98d] nova delete lxc-instance umounts the wrong rootfs (LP: #971621)
    - [09217ab] Block storage connections are NOT restored on system reboot
      (LP: #1036902)
    - [d9577ce] CVE-2012-3361 not fully addressed (LP: #1031311)
    - [e8ef050] pycrypto is unused and the existing code is potentially insecure
      to use (LP: #1033178)
    - [3b4ac31] cannot umount guestfs (LP: #1013689)
    - [f8255f3] qpid_heartbeat setting in ineffective (LP: #1030430)
    - [413c641] Deallocation of fixed IP occurs before security group refresh
      leading to potential security issue in error / race conditions
      (LP: #1021352)
    - [219c5ca] Race condition in network/deallocate_for_instance() leads to
      security issue (LP: #1021340)
    - [f2bc403] cleanup_file_locks does not remove stale sentinel files
      (LP: #1018586)
    - [4c7d671] Deleting Flavor currently in use by instance creates error
      (LP: #994935)
    - [7e88e39] nova testsuite errors on newer versions of python-boto (e.g.
      2.5.2) (LP: #1027984)
    - [80d3026] NoMoreFloatingIps: Zero floating ips available after repeatedly
      creating and destroying instances over time (LP: #1017418)
    - [4d74631] Launching with source groups under load produces lazy load error
      (LP: #1018721)
    - [08e5128] API 'v1.1/{tenant_id}/os-hosts' does not return a list of hosts
      (LP: #1014925)
    - [801b94a] Restarting nova-compute removes ip packet filters (LP: #1027105)
    - [f6d1f55] instance live migration should create virtual_size disk image
      (LP: #977007)
    - [4b89b4f] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [6e873bc] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [7b215ed] Use default qemu-img cluster size in libvirt connection driver
    - [d3a87a2] Listing flavors with marker set returns 400 (LP: #956096)
    - [cf6a85a] nova-rootwrap hardcodes paths instead of using
      /sbin:/usr/sbin:/usr/bin:/bin (LP: #1013147)
    - [2efc87c] affinity filters don't work if scheduler_hints is None
      (LP: #1007573)
  ...

Read more...

Changed in nova (Ubuntu Precise):
status: Confirmed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Thierry Carrez (ttx) on 2012-09-27
Changed in nova:
milestone: folsom-3 → 2012.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers