Quota exceeded for instances

[Dolph Mathews]

Log: http://logs.openstack.org/31559/2/check/gate-tempest-devstack-vm-quantum/27194/

There are a lot of failed tempest tests in this log, but the ones that stand out are regarding instance quota:

======================================================================
ERROR: test suite for <class 'tempest.api.volume.test_volumes_actions.VolumesActionsTest'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 208, in run
    self.setUp()
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 291, in setUp
    self.setupContext(ancestor)
  File "/usr/lib/python2.7/dist-packages/nose/suite.py", line 314, in setupContext
    try_run(context, names)
  File "/usr/lib/python2.7/dist-packages/nose/util.py", line 478, in try_run
    return func()
  File "/opt/stack/new/tempest/tempest/api/volume/test_volumes_actions.py", line 37, in setUpClass
    cls.flavor_ref)
  File "/opt/stack/new/tempest/tempest/services/compute/json/servers_client.py", line 81, in create_server
    resp, body = self.post('servers', post_body, self.headers)
  File "/opt/stack/new/tempest/tempest/common/rest_client.py", line 266, in post
    return self.request('POST', url, headers, body)
  File "/opt/stack/new/tempest/tempest/common/rest_client.py", line 394, in request
    resp, resp_body)
  File "/opt/stack/new/tempest/tempest/common/rest_client.py", line 455, in _error_checker
    raise exceptions.OverLimit(resp_body)
OverLimit: Quota exceeded
Details: Quota exceeded
Details: {u'overLimit': {u'message': u'Quota exceeded for instances: Requested 1, but already used 10 of 10 instances', u'code': 413, u'retryAfter': u'0'}}

[Sean Dague (sdague)]

So I thought we fixed that in cinder. This is really a cinder issue, not a tempest issue. So assigning there.

[Sean Dague (sdague)]

It looks like we are blowing past our servers quota, which is kind of amazing if true. It would be good for someone to dig through the logs to figure out where we are leaking servers.

[Attila Fazekas (afazekas)]

Looks like all server creation is failed.

https://bugs.launchpad.net/nova/+bug/1185834 is the root cause of the server boot issues.
I see many Similar KeyErrors in the log.

[Dolph Mathews (dolph)]

Agree with Attila- this is definitely at least a partial dupe of bug 1185834, but I hadn't seen the quota errors in my experience with bug 1185834.

[Attila Fazekas (afazekas)]

The quota issues has two possible sources:

If the server creation failes in setUpClass it not always be deleted.
If the creation fails in other cases does not a waited for deletion in every case.

I decrease the priority to low, the fail code path is less important at the moment.