OpenStack Compute (nova)

OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded

Bug #1349268 reported by Liyingjun
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Thang Pham
OpenStack Compute (nova) 2014.2 "juno"

Bug Description

The instance will be ERROR when booting instance from volume, if the volume quota is not enough. And there is even no useful error message to show to the user. Following is the related nova-compute.log:

2014-07-27 17:56:19.372 17060 ERROR nova.compute.manager [req-4e876b97-be8a-486b-98e2-7d707266755d 98fa3fd418914a9288b5560e1bb6944e 5254621adfd949a9a3b975f68119e269] [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] Instance failed block device setup
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] Traceback (most recent call last):
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1690, in _prep_block_device
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] self.driver, self._await_block_device_map_created))
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 363, in attach_block_devices
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] map(_log_and_attach, block_device_mapping)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 361, in _log_and_attach
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] bdm.attach(*attach_args, **attach_kwargs)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 311, in attach
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] '', '', image_id=self.image_id)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 303, in create
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] item = cinderclient(context).volumes.create(size, **kwargs)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volumes.py", line 187, in create
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] return self._create('/volumes', body, 'volume')
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 153, in _create
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] resp, body = self.api.client.post(url, body=body)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 209, in post
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] return self._cs_request(url, 'POST', **kwargs)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 173, in _cs_request
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] **kwargs)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 156, in request
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] raise exceptions.from_response(resp, body)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc] OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded (HTTP 413) (Request-ID: req-07dcc4c4-182f-4d73-b054-806f31cb7e71)
2014-07-27 17:56:19.372 17060 TRACE nova.compute.manager [instance: 2a124872-3332-4f54-bb28-f0a96a7ed7bc]
2014-07-27 17:56:19.693 17060 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Block Device Mapping is Invalid.
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 282, in decorated_function
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 268, in decorated_function
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 335, in decorated_function
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 311, in decorated_function
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2077, in run_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher do_run_instance()
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2076, in do_run_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher legacy_bdm_in_spec)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in _run_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher notify("error", fault=e) # notify that build failed
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1193, in _run_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher instance, image_meta, legacy_bdm_in_spec)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1347, in _build_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher LOG.exception(msg, instance=instance)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1304, in _build_instance
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher context, instance, bdms)
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1707, in _prep_block_device
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher raise exception.InvalidBDM()
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher InvalidBDM: Block Device Mapping is Invalid.
2014-07-27 17:56:19.693 17060 TRACE oslo.messaging.rpc.dispatcher

See original description
Tags: compute
Liyingjun (liyingjun)
description: updated
description: updated
Tracy Jones (tjones-i)
tags: added: compute
Thang Pham (thang-pham)
Changed in nova:
assignee: nobody → Thang Pham (thang-pham)
importance: Undecided → Low
Revision history for this message
Thang Pham (thang-pham) wrote :

I reproduced the error, and I am able to see an explanation of why it failed in n-cpu.log:

2014-08-20 21:54:16.115 DEBUG nova.compute.utils [req-0db5b860-2e6a-41d7-bb19-fa9f069cdc64 demo demo] [in
stance: 6880fd09-a82a-4965-9b0b-27ded82093d0] Build of instance 6880fd09-a82a-4965-9b0b-27ded82093d0 abor
ted: Failure prepping block device. from (pid=5906) notify_about_instance_usage /opt/stack/nova/nova/comp
ute/utils.py:306
2014-08-20 21:54:16.115 ERROR nova.compute.manager [req-0db5b860-2e6a-41d7-bb19-fa9f069cdc64 demo demo] [
instance: 6880fd09-a82a-4965-9b0b-27ded82093d0] Build of instance 6880fd09-a82a-4965-9b0b-27ded82093d0 ab
orted: Failure prepping block device.

However, it is a DEBUG message. Are you suggesting it be set to an error or warning log?

Revision history for this message
Thang Pham (thang-pham) wrote :

Also, looking the trace logs you have above, I no longer see "OverLimit: VolumeLimitExceeded" exceptions. Which version of nova are you using? I tried my tests above using the latest in the nova master branch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/115852

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/115852
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=26871aa60ba663d56951fcd449167bfadbe01522
Submitter: Jenkins
Branch: master

commit 26871aa60ba663d56951fcd449167bfadbe01522
Author: Thang Pham <email address hidden>
Date: Wed Aug 20 22:43:14 2014 -0400

    Log warning message if volume quota is exceeded

    It would be useful to the user if a warning message is logged
    when the volume quota is exceeded instead of a debug message.
    This would allow the root cause to be more easily determined
    instead of turning on the debug option.

    Change-Id: I19dc078cbacf8f39480e8a7f4c4edb764e4696eb
    Closes-Bug: #1349268

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.