[DB] Neutron quota request implementation can end in a lock status

[Rodolfo Alonso]

Neutron quota request implementation can end in a DB lock status. The quota is assigned per resource (port, network, security group, etc.) and per project. When a request is done, a DB lock is set for this (resource, project) tuple. This lock in the DB engine to lock this tuple in all workers of all API servers.

That implies there is a a bottleneck when a high number of requests arrive to the API at the same time. If the number of requests exceeds the number of resources processes, the DB locked transactions will increase indefinitely. This can be seen in the DB executing:

  $ mysql -e "show processlist;" | egrep "reservations|quotausages"

The query used by Neutron to lock this (resource, project) tuple is:

    UPDATE quotausages SET dirty=1 WHERE quotausages.project_id = <project_id> \
      AND quotausages.resource = <resource_type>

An improved quota system should be implemented that allow parallel resource request and avoids this DB lock status.

NOTE: please check [2][3]. "Neutron does not enforce quotas in such a way that a quota violation like this could never occur". That means even with this restrictive DB locking method, resource overcommit is possible.

[1]https://github.com/openstack/neutron/blob/b4812af4ee3cd651b0b03d5f90e71e8201ccfed7/neutron/objects/quota.py#L150
[2]https://bugzilla.redhat.com/show_bug.cgi?id=1884455#c2
[3]https://bugs.launchpad.net/neutron/+bug/1862050/comments/5

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1955661

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Open discussion in [1]. In [2] I propose an alternative quota system that drops the concept of resource "locks" and relays only on the DB current information.

I'll propose this as a RFE in the next Neutron meeting.

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1955661
[2]https://bugzilla.redhat.com/show_bug.cgi?id=1955661#c2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/790060

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/790060
Committed: https://opendev.org/openstack/neutron/commit/e135a8221dba3beac4047ca9351bdfe600fcf01a
Submitter: "Zuul (22348)"
Branch: master

commit e135a8221dba3beac4047ca9351bdfe600fcf01a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu May 6 12:48:05 2021 +0000

    New Quota driver ``DbQuotaNoLockDriver``

    This new quota driver, ``DbQuotaNoLockDriver``, does not create a lock
    per (resource, project_id) but retrieves the instant (resource,
    project_id) usage and the current (resource, project_id) reservations.
    If the requested number of resources fit the available quota, a new
    ``Reservation`` register is created with the amount of units requested.

    All those operations are done inside a DB transaction context. That
    means the amount of resources and reservations is guaranteed inside
    this transaction (depending on the DB backend isolation level defined)
    and the new reservation created will not clash with other DB transation.
    That will guarantee the number of resources and instant reservations
    never exceed the quota limits defined for this (resource, project_id).

    NOTES:
    - This change tries to be as unobtrusive as possible. The new driver
      uses the same ``DbQuotaDriver`` dabatase tables (except for
      ``QuotaUsage``) and the same Quota engine API, located in
      ``neutron.quota``. However, the Quota engine resources implements some
      particular API actions like "dirty", that are not used in the new
      driver.
    - The Pecan Quota enforcement hooks,
      ``neutron.pecan_wgsi.hooks.quota_enforcement``, execute actions like
      "resync", "mark_resources_dirty" or "set_resources_dirty", that has
      no meaning in the new driver.
    - The isolation between the Quota engine and the Pecan hook, and the
      driver itself is not clearly defined. A refactor of the Quota engine,
      Quota service, Quota drivers and a common API between the driver and
      the engine is needed.
    - If ``DbQuotaDriver`` is deprecated, ``CountableResource`` and
      ``TrackedResource`` will be joined in a single class. This resource
      class will have a count method (countable) or a hard dependency on a
      database table (tracked resource). The only difference will be the
      "count" method implementation.

    Closes-Bug: #1926787

    Change-Id: I4f98c6fcd781459fd7150aff426d19c7fdfa98c1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.