gate-neutron-lbaasv1-dsvm-api gate broken on Neutron LBaaS: fail to find a quota

[Victor Stinner]

The gate-neutron-lbaasv1-dsvm-api gate job of Neutron LBaaS looks to always fail on test_quotas. Example of a failure.

For example, the gate failed on the approve "Updated from global requirements" change: https://review.openstack.org/#/c/307761/

Console logs: http://logs.openstack.org/61/307761/1/check/gate-neutron-lbaasv1-dsvm-api/69872dd/console.html

neutron_lbaas.tests.tempest.v1.api.admin.test_quotas.QuotasTest.test_lbaas_quotas[gate]
---------------------------------------------------------------------------------------

Traceback (most recent call last):
  File "neutron_lbaas/tests/tempest/lib/services/network/json/network_client.py", line 330, in reset_quotas
    resp, body = self.delete(uri)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 290, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 642, in request
    resp, resp_body)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 695, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Quota for tenant ce7b2ca707c6479f88b67e312d3764f2 could not be found.', u'detail': u'', u'type': u'TenantQuotaNotFound'}

neutron_lbaas.tests.tempest.v1.api.admin.test_quotas.QuotasTest.test_quotas[gate]
---------------------------------------------------------------------------------

Traceback (most recent call last):
  File "neutron_lbaas/tests/tempest/lib/services/network/json/network_client.py", line 330, in reset_quotas
    resp, body = self.delete(uri)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 290, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 642, in request
    resp, resp_body)
  File "/opt/stack/new/neutron-lbaas/.tox/apiv1/local/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 695, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Quota for tenant 89128f4ca5aa415db790b0ed30d8b24e could not be found.', u'detail': u'', u'type': u'TenantQuotaNotFound'}

[Victor Stinner (vstinner)]

It looks like the bug started to occur today: https://review.openstack.org/#/c/306591/ got probably one of the first failure. Gerrit tells me "12:17 AM" (today). The check passed yesterday (April 19), but failed today (April 20).

[Victor Stinner (vstinner)]

The latest commit in Tempest occurred two days ago (April 18). https://github.com/openstack/tempest/commits/master

Same for Neutron LBaaS: latest commit occurred two days ago (April 18): https://github.com/openstack/neutron-lbaas/commits/master

Delta in pip-freeze:

--e git+https://git.openstack.org/openstack/keystone@23bb657369292cab3203c046a0a186df89fa1576#egg=keystone
+-e git+https://git.openstack.org/openstack/keystone@e380a3c005aa1c045bc15ef884edf3d0e20032f7#egg=keystone

--e git+https://git.openstack.org/openstack/neutron@8f33cbeb4264d059ce4daa933c5381ab54e34439#egg=neutron
+-e git+https://git.openstack.org/openstack/neutron@21a2dbc25d2eeeae43faf6c7e5ebd2e4f7fd2be2#egg=neutron

--e git+https://git.openstack.org/openstack/neutron-lbaas@244e82983cc8ab4530d258fdc31d91fbc791b310#egg=neutron_lbaas
+-e git+https://git.openstack.org/openstack/neutron-lbaas@1d1bf61ff7ffa348f3f5e6c9acf14b7c7b39bcd8#egg=neutron_lbaas

--e git+https://git.openstack.org/openstack/nova@654181beeb6e8ea813b86b8dff05bc084c9e9bce#egg=nova
+-e git+https://git.openstack.org/openstack/nova@e276284791be5ee203a11283887d5cdfd2bd064e#egg=nova

IMHO the following Neutron change is a good candidate, it was merged at "Tue Apr 19 15:15:29 2016 +0000":

commit 915352725296de45a74ffd2ea1edc4c498eb4a8f
Author: reedip <email address hidden>
Date: Thu Feb 4 17:36:32 2016 +0900

    Return a 404 on quota delete if entry not found

    If a tenant quota is not found, the delete operation should fail
    with a 404 error. It currently returns 204 even if the delete
    operation actually does not delete anything which is inconsistent
    with other delete operations in the Neutron API.

    APIImpact

    Change-Id: I1cd91b5e06bd17f9aac97bba71228f2e5c48879b
    Closes-Bug: 1307506
    Co-Authored-By: Salvatore Orlando <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/308384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/308384
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=c5cbd394e1aca10d915272c3e787129949e9f6a2
Submitter: Jenkins
Branch: master

commit c5cbd394e1aca10d915272c3e787129949e9f6a2
Author: Victor Stinner <email address hidden>
Date: Wed Apr 20 16:46:27 2016 +0200

    Fix test_quotas

    The change I1cd91b5e06bd17f9aac97bba71228f2e5c48879b modified the
    delete_tenant_quota() function: it now raises a NotFound error if
    delete failed.

    Fix the test: catch and ignore the NotFound exception. The test
    removes the quota by reseting quotas of the tenant.

    Change-Id: I87df52474014c4bceff8124a627f829f6b625ce0
    Closes-Bug: #1572592

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron-lbaas 9.0.0.0b1 development milestone.