Wrong calculation of quotas

Bug #1623390 reported by Andrey Kurilin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned

Bug Description

Neutron has rally non-voting job. It is red now.
Rally report: http://logs.openstack.org/44/367744/7/check/gate-rally-dsvm-neutron-neutron/b3af93f/rally-plot/results.html.gz#/NeutronNetworks.create_and_list_networks/failures

Scenario:
- [pre-step] create new test tenant
- [pre-step] set quotas for new tenant to allow create 100 networks (neutron.quotas.update networks=100)
- [repeat 100 times] create and list networks

Expected result: all 100 iterations are finished successfully == it is possible to create N networks if networks quotas is set to N
Actual result: The last iteration is failed == it is possible to create N-1 networks if networks quotas is set to N

Sergey Belous (sbelous)
Changed in neutron:
assignee: nobody → Sergey Belous (sbelous)
Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :

I see the error, but neither logstash nor grafana confirm it. Failure rate is not 100% [1]

[1] http://grafana.openstack.org/dashboard/db/neutron-failure-rate?from=1468664457637&to=1473848457637

Revision history for this message
Hirofumi Ichihara (ichihara-hirofumi) wrote :
Revision history for this message
Dariusz Smigiel (smigiel-dariusz) wrote :
Revision history for this message
Kevin Benton (kevinbenton) wrote :

This is actually going to be expected behavior right now with the way the quota engine works. See the commit message in https://review.openstack.org/371739 for an explanation.

Changed in neutron:
status: New → Won't Fix
Revision history for this message
Kevin Benton (kevinbenton) wrote :

Marked as "Won't fix" because we probably won't be able to change the quota calculation behavior any time soon. That patch will adjust the rally job though to expect this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/371739
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8
Submitter: Jenkins
Branch: master

commit 933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8
Author: Kevin Benton <email address hidden>
Date: Fri Sep 16 11:39:14 2016 -0600

    Add to rally quotas to handle worst case quota race

    Quoting the quota devref:
    """
    For a reservation to be successful, the total amount of resources requested,
    plus the total amount of resources reserved, plus the total amount of resources
    already stored in the database should not exceed the project's quota limit.
    """

    This means that in the absolute worst case scenario with 20 concurrent
    workers, 19 could have made reservations, committed resources, but not
    yet cleared their reservation. Because of the outstanding reservation
    and the resources created by the 19 workers, they will all be
    double-counted until their reservation is cleared (or it expires).

    This adjusts the rally scenarios to handle the double-count for
    concurrency.

    Related-Bug: #1623390
    Change-Id: I4808a92e7e6067aeeb62fc3b3d7f7ac71b179c44

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/374532

Sergey Belous (sbelous)
Changed in neutron:
assignee: Sergey Belous (sbelous) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/374532
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cbd7780214cf872439dc9eca732040c66fadfc0b
Submitter: Jenkins
Branch: stable/newton

commit cbd7780214cf872439dc9eca732040c66fadfc0b
Author: Kevin Benton <email address hidden>
Date: Fri Sep 16 11:39:14 2016 -0600

    Add to rally quotas to handle worst case quota race

    Quoting the quota devref:
    """
    For a reservation to be successful, the total amount of resources requested,
    plus the total amount of resources reserved, plus the total amount of resources
    already stored in the database should not exceed the project's quota limit.
    """

    This means that in the absolute worst case scenario with 20 concurrent
    workers, 19 could have made reservations, committed resources, but not
    yet cleared their reservation. Because of the outstanding reservation
    and the resources created by the 19 workers, they will all be
    double-counted until their reservation is cleared (or it expires).

    This adjusts the rally scenarios to handle the double-count for
    concurrency.

    Related-Bug: #1623390
    Change-Id: I4808a92e7e6067aeeb62fc3b3d7f7ac71b179c44
    (cherry picked from commit 933af3fdac4d7d8dd8bf2b419c87b9dc27d69af8)

tags: added: in-stable-newton
tags: added: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers