[RFE] Neutron quota change should check available existing resources

[Rodolfo Alonso]

Neutron quota change should check available existing resources. This is done, for example, in Nova. When a quota resource limit is changed, the available resource count is checked first. If the new quota upper limit (lower than the previous one) is lower than the amount of resources in use, the quota driver should raise an exception.

This RFE implies a change in the Neutron quota current behaviour. Some users are expecting the new quota limit to be applied, regardless of being lower than the current resource usage.

However, other users (Octavia) expect the quota driver to fail when lowering the quota limit under the existing resource usage. My recommendation is to use a config knob to decide the behaviour of the quota driver; by default, the current behaviour will prevail.

Bugzilla reference: https://bugzilla.redhat.com/show_bug.cgi?id=1980728

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/801470

[Michael Johnson (johnsom)]

Actually, Octavia behaves the same as neutron and expects neutron to behave the way it does now.

A similar RFE was opened against Octavia. It also allows you to set a quota lower than the current in-use resources to allow an operator to "drain" a project of resources.

[Brian Haley (brian-haley)]

I am of the opinion this is not a bug. Lowering the quota for a resource below what exists should be allowed, it will just make sure that the new quota is used in future calls. For example, if it generates an error how is the user to change the value without deleting resources?

I will agree that components have different responses here, so if Neutron (and Octavia) is somehow the only one behaving differently then I could see allowing it via a config value, but changing it without that will break the API contract.

Also, Octavia behaves similarly to Neutron - the quota can be lowered without an error.

[Brian Haley (brian-haley)]

So maybe a better example is this.

Current behavior:

1) Quota for networks is set to 10
2) Project A creates 10 networks
3) Admin changes quota to 5 networks
4) Project B tries to create 10 networks and fails after 5 (good)

Proposed behavior:

1) Quota for networks is set to 10
2) Project A creates 10 networks
3) Admin changes quota to 5 networks, which fails
4) Project B tries to create 10 networks and succeeds (bad)

So now there are two projects with 10 networks instead of just one. So the admin is never able to enforce the new quota of 5 networks until all users reduce to that number, therefore any new project can create 10 networks.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Octavia is not the only Neutron consumer.

The proposal I'm making do NOT change the default behaviour. What I'm proposing is to make it configurable, always keeping the default one.

@Brian: you description of "Proposed behavior" will depend on the consumer. If the consumer does not check the error in step 3, that's its fault.

Regards.

[Brian Haley (brian-haley)]

Rodolfo - yes, understood. I guess my point is that if you can't lower the max quota below the max in-use value, then others can increase their usage too, and you'll never be able to enforce it. That doesn't seem like the intention.

[Slawek Kaplonski (slaweq)]

Let's discuss that RFE on the drivers meeting next Friday (30.07.2021)

[Slawek Kaplonski (slaweq)]

We discussed that RFE on today's drivers meeting https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.2021-08-06-14.03.log.html#l-14 and we all agreed that current behavior is actually a feature and we shouldn't change it. It also aligns with the comment #6 from Brian.
So we decided to reject that RFE.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I'm reopening this RFE again and I hope next Friday I'll have time to present it in the driver's meeting.

The goal of this RFE is NOT to change the default API behaviour but to provide an alternative one. I initially presented several alternatives but none of them were good enough, due to how quota engine is loaded and the lack of defined API (it is created during the module load).

What I'm proposing is to add a new parameter to the CLI that triggers the check (and possible rejection) of the new quota limit. To provide some context, Nova do not allow to lower a quota limit below the current usage. However it is possible to override a limit using the CLI parameter "force" [1]. The proposal is just the opposite to what Nova implements: a new parameter (e.g.: "check-limit") that will be passed in the CLI command. if this parameter is present, the Neutron quota engine will check the current quota usage and limit and will reject, if needed, the quota update (same as in Nova).

Regards.

[1]https://github.com/openstack/python-openstackclient/blob/ed87f7949ef1ef580ed71b9820e16823c0466472/openstackclient/common/quota.py#L533-L536

[Slawek Kaplonski (slaweq)]

Let's discuss it again in the next drivers meeting on Friday 20.08.2021

[Brian Haley (brian-haley)]

Hi Rodolfo - can you comment more on what the new behavior would be? It's hard to tell if --force would be required to get the current behavior and it would fail otherwise. Asking since I won't be able to attend the meeting Friday. Thanks.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Brian:

First of all, let me highlight that what I'm proposing will NOT change the default behaviour. That means, any user script using "openstack" CLI will be affected.

What I'm going to propose on Friday is to add to the CLI a new parameter for "network" commands. To document this parameter I provided [1] as an example: a parameter only for "compute" commands that modifies the command behaviour.

**BUT** this is an example of implementation. What I'm proposing is the opposite: a new CLI parameter, only for "network" commands, that will enforce the quota check, something we currently don't do. This parameter could be called "check-quota" (for example, I'm very bad at naming):
- If this new parameter is added to the command, the Neutron server will check the new quota limit provided and fail if needed.
- If this new parameter is not provided, we'll work as usual; that means we'll be able to lower the quota limit without any check (as is now).

Again, I'm not proposing a default behaviour change. What I'm proposing is adding a new parameter that will trigger the quota limit check.

NOTE: this is the default behaviour in Nova. Nova cannot lower a resource limit (VCPUS for example) if VMs are using more. However, Nova is able to lower this limit using "--force" [1].

Regards.

[1]https://github.com/openstack/python-openstackclient/blob/ed87f7949ef1ef580ed71b9820e16823c0466472/openstackclient/common/quota.py#L533-L536

[Slawek Kaplonski (slaweq)]

We discussed that RFE again on today's drivers meeting: https://meetings.opendev.org/meetings/neutron_drivers/2021/neutron_drivers.2021-08-27-14.00.log.html#l-61
and we agreed to accept it. We can add something like "--check-limit" option to our API for quotas.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/807876

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/807876
Committed: https://opendev.org/openstack/neutron-lib/commit/14df80519c111a896f57886dc5db93284682dbba
Submitter: "Zuul (22348)"
Branch: master

commit 14df80519c111a896f57886dc5db93284682dbba
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 8 13:42:43 2021 +0000

    Add API shim extension "quota-check-limit"

    This extension indicates if Neutron supports checking the resource
    usage before applying a new quota limit.

    Change-Id: I94baf069624da4e4009eb8c0820f91b4d9e6738f
    Related-Bug: #1936408

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/801470
Committed: https://opendev.org/openstack/neutron/commit/5a7a8db0d87c2027287e8fb074ee0f681e6369bb
Submitter: "Zuul (22348)"
Branch: master

commit 5a7a8db0d87c2027287e8fb074ee0f681e6369bb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Aug 26 17:55:35 2021 +0000

    Check quota limits

    When "check_limit" parameter is passed in a quota update request,
    the Neutron server checks the current resource usage before updating
    the quota limit. If the new quota limit is below the resource usage,
    an exception is raised.

    This parameter was added in [1][2].

    [1]https://review.opendev.org/c/openstack/openstacksdk/+/806254
    [2]https://review.opendev.org/c/openstack/python-openstackclient/+/806016

    Closes-Bug: #1936408

    Change-Id: I5a6fb65694498dd7d8f403ea04dc1fe72b8c938d

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/819414

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/819414
Committed: https://opendev.org/openstack/neutron/commit/42f52a663b7107670ad94697ebdd9d20bfc38d6b
Submitter: "Zuul (22348)"
Branch: master

commit 42f52a663b7107670ad94697ebdd9d20bfc38d6b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Nov 26 09:50:26 2021 +0000

    Add "quota-check-limit" extension to OVN extension list

    Added "quota-check-limit" extension to the OVN ML2 extension list.
    This extension is also added to the NETWORK_API_EXTENSIONS list.

    Related-Bug: #1936408
    Change-Id: Ifbe49146555c8103efe9cd3979a120697a167d9d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.