[DB] Neutron quota request implementation can end in a lock status

Bug #1926787 reported by Rodolfo Alonso
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released
Rodolfo Alonso

Bug Description

Neutron quota request implementation can end in a DB lock status. The quota is assigned per resource (port, network, security group, etc.) and per project. When a request is done, a DB lock is set for this (resource, project) tuple. This lock in the DB engine to lock this tuple in all workers of all API servers.

That implies there is a a bottleneck when a high number of requests arrive to the API at the same time. If the number of requests exceeds the number of resources processes, the DB locked transactions will increase indefinitely. This can be seen in the DB executing:

  $ mysql -e "show processlist;" | egrep "reservations|quotausages"

The query used by Neutron to lock this (resource, project) tuple is:

    UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
      AND quotausages.resource = <resource_type>

An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status.

NOTE: please check [2][3]. "Neutron does not enforce quotas in such a way that a quota violation like this could never occur". That means even with this restrictive DB locking method, resource overcommit is possible.


Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

Tags: db
Changed in neutron:
importance: Undecided → Wishlist
description: updated
Changed in neutron:
status: New → Confirmed
tags: added: db
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Open discussion in [1]. In [2] I propose an alternative quota system that drops the concept of resource "locks" and relays only on the DB current information.

I'll propose this as a RFE in the next Neutron meeting.


Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/790060

Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/790060
Committed: https://opendev.org/openstack/neutron/commit/e135a8221dba3beac4047ca9351bdfe600fcf01a
Submitter: "Zuul (22348)"
Branch: master

commit e135a8221dba3beac4047ca9351bdfe600fcf01a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu May 6 12:48:05 2021 +0000

    New Quota driver ``DbQuotaNoLockDriver``

    This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
    per (resource, project_id) but retrieves the instant (resource,
    project_id) usage and the current (resource, project_id) reservations.
    If the requested number of resources fit the available quota, a new
    ``Reservation`` register is created with the amount of units requested.

    All those operations are done inside a DB transaction context. That
    means the amount of resources and reservations is guaranteed inside
    this transaction (depending on the DB backend isolation level defined)
    and the new reservation created will not clash with other DB transation.
    That will guarantee the number of resources and instant reservations
    never exceed the quota limits defined for this (resource, project_id).

    - This change tries to be as unobtrusive as possible. The new driver
      uses the same ``DbQuotaDriver`` dabatase tables (except for
      ``QuotaUsage``) and the same Quota engine API, located in
      ``neutron.quota``. However, the Quota engine resources implements some
      particular API actions like "dirty", that are not used in the new
    - The Pecan Quota enforcement hooks,
      ``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
      "resync", "mark_resources_dirty" or "set_resources_dirty", that has
      no meaning in the new driver.
    - The isolation between the Quota engine and the Pecan hook, and the
      driver itself is not clearly defined. A refactor of the Quota engine,
      Quota service, Quota drivers and a common API between the driver and
      the engine is needed.
    - If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
      ``TrackedResource`` will be joined in a single class. This resource
      class will have a count method (countable) or a hard dependency on a
      database table (tracked resource). The only difference will be the
      "count" method implementation.

    Closes-Bug: #1926787

    Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron

This issue was fixed in the openstack/neutron release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.