Wrong calculation of quotas

[Andriy Kurilin]

Neutron has rally non-voting job. It is red now.
Rally report: http://logs.openstack.org/44/367744/7/check/gate-rally-dsvm-neutron-neutron/b3af93f/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/failures

Scenario:
- [pre-step] create new test tenant
- [pre-step] set quotas for new tenant to allow create 100 networks (neutron.quotas.update networks=100)
- [repeat 100 times] create and list networks

Expected result: all 100 iterations are finished successfully == it is possible to create N networks if networks quotas is set to N
Actual result: The last iteration is failed == it is possible to create N-1 networks if networks quotas is set to N

[Dariusz Smigiel (smigiel-dariusz)]

I see the error, but neither logstash nor grafana confirm it. Failure rate is not 100% [1]

[1] http://grafana.openstack.org/dashboard/db/neutron-failure-rate?from=1468664457637&to=1473848457637

[Hirofumi Ichihara (ichihara-hirofumi)]

5 hits in 24h

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Quota%20exceeded%20for%20resources%3A%20%5B'network'%5D%5C%22

However, I'm not sure whether neutron bug or rally bug.

[Dariusz Smigiel (smigiel-dariusz)]

Hmm.. 24h just for Neutron shows 31 failures [1] while for all projects (link by Hirofumi) it's over 110. 3 tempest failures [2] but 53 tests shown with this message [3]. And, funny thing, there is even 1 success for neutron [4]
All in 24h.

[1] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Quota%20exceeded%20for%20resources%3A%20%5B'network'%5D%5C%22%20AND%20project%3A%20%5C%22openstack%2Fneutron%5C%22%20AND%20build_name%3A%20%5C%22gate-rally-dsvm-neutron-neutron%5C%22%20AND%20build_status%3A%20%5C%22FAILURE%5C%22
[2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Quota%20exceeded%20for%20resources%3A%20%5B'network'%5D%5C%22%20AND%20project%3A%20%5C%22openstack%2Ftempest%5C%22%20AND%20build_status%3A%20%5C%22FAILURE%5C%22
[3] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Quota%20exceeded%20for%20resources%3A%20%5B'network'%5D%5C%22%20AND%20project%3A%20%5C%22openstack%2Ftempest%5C%22%20
[4] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22Quota%20exceeded%20for%20resources%3A%20%5B'network'%5D%5C%22%20AND%20project%3A%20%5C%22openstack%2Fneutron%5C%22%20AND%20build_status%3A%20%5C%22SUCCESS%5C%22

[Kevin Benton (kevinbenton)]

This is actually going to be expected behavior right now with the way the quota engine works. See the commit message in https://review.openstack.org/371739 for an explanation.

[Kevin Benton (kevinbenton)]

Marked as "Won't fix" because we probably won't be able to change the quota calculation behavior any time soon. That patch will adjust the rally job though to expect this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/371739
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8
Submitter: Jenkins
Branch: master

commit 933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8
Author: Kevin Benton <email address hidden>
Date: Fri Sep 16 11:39:14 2016 -0600

    Add to rally quotas to handle worst case quota race

    Quoting the quota devref:
    """
    For a reservation to be successful, the total amount of resources requested,
    plus the total amount of resources reserved, plus the total amount of resources
    already stored in the database should not exceed the project's quota limit.
    """

    This means that in the absolute worst case scenario with 20 concurrent
    workers, 19 could have made reservations, committed resources, but not
    yet cleared their reservation. Because of the outstanding reservation
    and the resources created by the 19 workers, they will all be
    double-counted until their reservation is cleared (or it expires).

    This adjusts the rally scenarios to handle the double-count for
    concurrency.

    Related-Bug: #1623390
    Change-Id: I4808a92e7e6067aeeb62fc3b3d7f7ac71b179c44

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/374532

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374532
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cbd7780214cf872439dc9eca732040c66fadfc0b
Submitter: Jenkins
Branch: stable/newton

commit cbd7780214cf872439dc9eca732040c66fadfc0b
Author: Kevin Benton <email address hidden>
Date: Fri Sep 16 11:39:14 2016 -0600

    Add to rally quotas to handle worst case quota race

    Quoting the quota devref:
    """
    For a reservation to be successful, the total amount of resources requested,
    plus the total amount of resources reserved, plus the total amount of resources
    already stored in the database should not exceed the project's quota limit.
    """

    This means that in the absolute worst case scenario with 20 concurrent
    workers, 19 could have made reservations, committed resources, but not
    yet cleared their reservation. Because of the outstanding reservation
    and the resources created by the 19 workers, they will all be
    double-counted until their reservation is cleared (or it expires).

    This adjusts the rally scenarios to handle the double-count for
    concurrency.

    Related-Bug: #1623390
    Change-Id: I4808a92e7e6067aeeb62fc3b3d7f7ac71b179c44
    (cherry picked from commit 933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8)