Functional tests are failing with over-quota errors on number of instances

[Matt Riedemann]

Seen here:

http://logs.openstack.org/41/435141/19/gate/gate-novaclient-dsvm-functional-neutron-ubuntu-xenial/894e11f/console.html#_2017-06-24_07_43_15_452547

This isn't specific to that test:

2017-06-24 07:43:15.452547 | 2017-06-24 07:43:15.452 | novaclient.exceptions.Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) (Request-ID: req-462a3601-792a-4184-941b-efcf6dc08cf4)
2017-06-24 07:43:15.453606 | 2017-06-24 07:43:15.453 |
2017-06-24 07:43:17.028370 | 2017-06-24 07:43:17.028 | {0} novaclient.tests.functional.v2.legacy.test_usage.TestUsageCLI.test_usage [1.605477s] ... FAILED
2017-06-24 07:43:17.030172 | 2017-06-24 07:43:17.029 |
2017-06-24 07:43:17.031396 | 2017-06-24 07:43:17.031 | Captured traceback:
2017-06-24 07:43:17.034342 | 2017-06-24 07:43:17.033 | ~~~~~~~~~~~~~~~~~~~
2017-06-24 07:43:17.035573 | 2017-06-24 07:43:17.035 | Traceback (most recent call last):
2017-06-24 07:43:17.037772 | 2017-06-24 07:43:17.036 | File "novaclient/tests/functional/v2/legacy/test_usage.py", line 37, in test_usage
2017-06-24 07:43:17.039170 | 2017-06-24 07:43:17.038 | self._create_server('some-server')
2017-06-24 07:43:17.040454 | 2017-06-24 07:43:17.040 | File "novaclient/tests/functional/base.py", line 479, in _create_server
2017-06-24 07:43:17.042225 | 2017-06-24 07:43:17.041 | nics=nics, **kwargs)
2017-06-24 07:43:17.043590 | 2017-06-24 07:43:17.043 | File "novaclient/v2/servers.py", line 1437, in create
2017-06-24 07:43:17.045230 | 2017-06-24 07:43:17.044 | **boot_kwargs)
2017-06-24 07:43:17.046705 | 2017-06-24 07:43:17.046 | File "novaclient/v2/servers.py", line 790, in _boot
2017-06-24 07:43:17.047980 | 2017-06-24 07:43:17.047 | return_raw=return_raw, **kwargs)
2017-06-24 07:43:17.049216 | 2017-06-24 07:43:17.048 | File "novaclient/base.py", line 361, in _create
2017-06-24 07:43:17.050501 | 2017-06-24 07:43:17.050 | resp, body = self.api.client.post(url, body=body)
2017-06-24 07:43:17.051751 | 2017-06-24 07:43:17.051 | File "/opt/stack/new/python-novaclient/.tox/functional/local/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 240, in post
2017-06-24 07:43:17.054131 | 2017-06-24 07:43:17.052 | return self.request(url, 'POST', **kwargs)
2017-06-24 07:43:17.055475 | 2017-06-24 07:43:17.055 | File "novaclient/client.py", line 83, in request
2017-06-24 07:43:17.056805 | 2017-06-24 07:43:17.056 | raise exceptions.from_response(resp, body, url, method)
2017-06-24 07:43:17.058478 | 2017-06-24 07:43:17.058 | novaclient.exceptions.Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) (Request-ID: req-058b10ff-96ab-4204-965b-2fcf8b2759ca)

Something isn't cleaning up properly which is making the tests go over quota since they are single-tenant.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22novaclient.exceptions.Forbidden%3A%20Quota%20exceeded%20for%20instances%3A%20Requested%201%2C%20but%20already%20used%2010%20of%2010%20instances%20(HTTP%20403)%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

It looks like something merged around 6/24 that tipped this over.

[Matt Riedemann (mriedem)]

I don't see any changes to novaclient in the last 7 days, so maybe we regressed something on the nova API side with the recent counting quotas work.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477217

[Matt Riedemann (mriedem)]

I don't see anything changed in nova or novaclient since 6/24 that could cause this, so I'm wondering if this devstack change is causing some problems:

https://review.openstack.org/#/c/436094/

[Rabi Mishra (rabi)]

We do have the same issue in the heat gate.

http://logs.openstack.org/93/477393/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial/b67f7ca/console.html#_2017-06-26_07_12_04_194467

[Rabi Mishra (rabi)]

I've proposed a patch to disable quota check in our jobs sometime back.

https://review.openstack.org/#/c/477368/

[Matt Riedemann (mriedem)]

I've confirmed that a revert of the devstack patch to fleetify conductor in devstack makes the novaclient functional tests pass again, so that's the root trigger:

https://review.openstack.org/#/c/477323/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/477548

[Matt Riedemann (mriedem)]

The devstack revert is approved: https://review.openstack.org/#/c/477436/

[Matt Riedemann (mriedem)]

This novaclient change will test the conductor fleet concept but with quotas in the API DB:

https://review.openstack.org/#/c/477551/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/477315

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/477436
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=20eeff813d21e86c1d529d5cbc4f7b0ad732e69d
Submitter: Jenkins
Branch: master

commit 20eeff813d21e86c1d529d5cbc4f7b0ad732e69d
Author: Matt Riedemann <email address hidden>
Date: Mon Jun 26 07:53:08 2017 +0000

    Revert "Fleetify nova conductor for N cells"

    This reverts commit f63aa021cf996b500b6a570c0114c490bb57461a.

    Change-Id: I541abe7311e5721d79e300fcb2a0b8ffdcd3c015
    Related-Bug: #1700364

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/477548
Reason: reverting the devstack change for multi-cell fixed the novaclient quotas failures

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/477217
Reason: I think we can abandon this since reverting the devstack multi-cell change fixed the issue - we can't do multi-cell until we have quotas in the API and the new counting quotas scheme.