Neutron hits "designateclient.exceptions.OverQuota: over_quota" when creating floating IP

Bug #1980673 reported by Nicolas Melot
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Unassigned

Bug Description

Posting here because the issue seems to be related to Designate quotas. Looks like Neutron is only a victim here.

OpenStack fails when creating a floating IP address with a DNS name and zone. Designate logs do not show anything interesting but neutron server reports "designateclient.exceptions.OverQuota: over_quota" when creating floating IP" and the CLI command outputs "Request Failed: internal server error while processing your request." and exists with non-zero value.

The floating IP is created but the new DNS name cannot be resolved. The floating IP cannot be removed until the CLI command to remove it is run again.

This fails when creating a floating IP within 10.18.75.x, but it works for addresses within 10.18.86.x. The second range has been working just fine to this day, and creating an IP address within this range works as expected.

The logs suggest that the project hits some quota for the reverse zone. However I fail to check how much resource under quota the project uses. Since the project has just been created (beside a few similar attempts that fails), I expect such usage to be near 0.

I cannot find any helpful direction on internet for a corner to look at with. I attached the logs I collected for neutron server and all designate services, as well as the command I run and their output.

This is OpenStack Xena running on Rocky 8.5-based nodes.

Revision history for this message
Nicolas Melot (nicmel) wrote :
affects: designate → neutron
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

https://review.opendev.org/c/openstack/neutron/+/848420 was proposed to fix the broken error handling in Neutron. You will still have to check why the quota for DNS zones is exceeded. Maybe you simply have more than 10 (which is the default quota IIRC) of them?

tags: added: dns
Revision history for this message
Nicolas Melot (nicmel) wrote :

Is there any way to check how much each tenant use of their DNS quota? I could not get the designate CLI client to work on Rocky 8.5. I checked directly in the database for zones and I found that the "service project" tenant has 10 reverse dns zones, which was the zone quota setting. After setting the zone quota to 20, then the operation worked as expected. Problem solved then.

I find odd that tenants are dependents on the "service project" tenant's quota to create their own records, because of the reverse dns zone that needs to be setup. Surely this is a potential denial of service weakness, by having one tenant to hog "service project" quotas by creating many records and reverse zones.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

The reverse zones are owned by the service project since sharing of zones between multiple projects isn't implemented in Designate, but the reverse zones can contain records from different tenants. Automated creation of reverse zones is a questionable feature anyway, since without proper delegation being set up by the deployer, they are pretty useless. Ideas and patches to improve this are welcome.

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Also the designate CLI client is indeed deprecated and no longer works. You should be able to see your quota limits with "openstack dns quota list", although it seems that there is a bug that makes this only possible for admins currently. The actual usage data isn't available there, you would have to list the actual resources and count them.

Changed in neutron:
status: New → In Progress
Revision history for this message
Nicolas Melot (nicmel) wrote :

That explains why I could not get that command working. Thank you for your help!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/848420
Committed: https://opendev.org/openstack/neutron/commit/8b0a34a5857ab2a1fbb9a05db9a01ec4173164c3
Submitter: "Zuul (22348)"
Branch: master

commit 8b0a34a5857ab2a1fbb9a05db9a01ec4173164c3
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Mon Jul 4 16:52:07 2022 -0700

    Report ExternalDNSOverQuota exception for zone quota error

    This bug was previously fixed for recordsets, but can also fail
    when creating zones.

    Closes-Bug: #1980673
    Related-Bug: #1923700
    Change-Id: Icef3eab6a27f3b8950a7877de0d387e668a067ad

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/848722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/848723

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/848723
Committed: https://opendev.org/openstack/neutron/commit/0eb8e2b21652bd1beacfdf6079baca660737625e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 0eb8e2b21652bd1beacfdf6079baca660737625e
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Mon Jul 4 16:52:07 2022 -0700

    Report ExternalDNSOverQuota exception for zone quota error

    This bug was previously fixed for recordsets, but can also fail
    when creating zones.

    Closes-Bug: #1980673
    Related-Bug: #1923700
    Change-Id: Icef3eab6a27f3b8950a7877de0d387e668a067ad
    (cherry picked from commit 8b0a34a5857ab2a1fbb9a05db9a01ec4173164c3)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/848722
Committed: https://opendev.org/openstack/neutron/commit/c31cc3d6e292616b0b7f7422ac66228c58e2b59e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit c31cc3d6e292616b0b7f7422ac66228c58e2b59e
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Mon Jul 4 16:52:07 2022 -0700

    Report ExternalDNSOverQuota exception for zone quota error

    This bug was previously fixed for recordsets, but can also fail
    when creating zones.

    Closes-Bug: #1980673
    Related-Bug: #1923700
    Change-Id: Icef3eab6a27f3b8950a7877de0d387e668a067ad
    (cherry picked from commit 8b0a34a5857ab2a1fbb9a05db9a01ec4173164c3)

tags: added: in-stable-yoga
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Just for completeness, this is the bug report for the "dns quota list" issue with a pending fix already: https://bugs.launchpad.net/designate/+bug/1980942

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.4.0

This issue was fixed in the openstack/neutron 19.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.2.0

This issue was fixed in the openstack/neutron 20.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.0.0.0rc1

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.