Can't create new floating IP when the pool of IPs is empty: ERROR (Forbidden): IP allocation over quota. (HTTP 403)

[Timur Nurlygayanov]

Note:
Reproduced on MOS 8.0 ISO #264
Looks like the root of the problem in incorrect quotas verification / floating IPs allocation

Steps To Reproduce:
1. Deploy OpenStack cluster with Neutron VLANs
2. Login to OpenStack controller node
3. Create many floating IPs and assign some of them to VMs
4. Disassociate floating IPs from VMs
5. Run on controller node to remove all floating IPs:
for i in `nova floating-ip-list | grep -e "- " | awk '{print $4}'`; do nova floating-ip-delete $i; done
6. Check that we haven't floating IPs anymore:
root@node-1:~# nova floating-ip-list
+----+----+-----------+----------+------+
| Id | IP | Server Id | Fixed IP | Pool |
+----+----+-----------+----------+------+
+----+----+-----------+----------+------+
7. Try to create floating IP:
nova floating-ip-create

Observed Result:
User can't create floating IPs in this cluster:
root@node-1:~# nova floating-ip-create
ERROR (Forbidden): IP allocation over quota. (HTTP 403) (Request-ID: req-a143bc8b-d68e-4737-b4a2-3b72e898a020)

We have no quotas for Floating IPs (we can successfully create new floating IPs on the first step, but can't on the last one):
root@node-1:~# nova quota-show
+-----------------------------+-------+
| Quota | Limit |
+-----------------------------+-------+
| instances | -1 |
| cores | -1 |
| ram | -1 |
| floating_ips | -1 |
| fixed_ips | -1 |
| metadata_items | -1 |
| injected_files | -1 |
| injected_file_content_bytes | -1 |
| injected_file_path_bytes | -1 |
| key_pairs | -1 |
| security_groups | -1 |
| security_group_rules | -1 |
| server_groups | -1 |
| server_group_members | -1 |
+-----------------------------+-------+

[Timur Nurlygayanov (tnurlygayanov)]

mysql> select * from quotausages;
+----------------------------------+----------------+-------+--------+----------+
| tenant_id | resource | dirty | in_use | reserved |
+----------------------------------+----------------+-------+--------+----------+
| 7ff88ebe754d49398171c9a441087ecc | floatingip | 0 | 50 | 0 |
| 7ff88ebe754d49398171c9a441087ecc | network | 0 | 2 | 0 |
| 7ff88ebe754d49398171c9a441087ecc | port | 0 | 4 | 0 |
| 7ff88ebe754d49398171c9a441087ecc | router | 0 | 1 | 0 |
| 7ff88ebe754d49398171c9a441087ecc | security_group | 0 | 4 | 0 |
| 7ff88ebe754d49398171c9a441087ecc | subnet | 0 | 2 | 0 |
| c0ea5042eed64c97b7b7731ed38636ac | security_group | 0 | 1 | 0 |
+----------------------------------+----------------+-------+--------+----------+
7 rows in set (0.00 sec)

mysql> select * from floatingips;
Empty set (0.00 sec)

[Roman Podoliaka (rpodolyaka)]

I checked the creation of floating IPs directly in Neutron (rather than relying on proxying done by nova-api):

root@node-1:~# neutron floatingip-create admin_floating_net
neutron Quota exceeded for resources: ['floatingip']

root@node-1:~# neutron quota-show
+---------------------+-------+
| Field | Value |
+---------------------+-------+
| floatingip | 50 |
| network | 10 |
| port | 50 |
| rbac_policy | -1 |
| router | 10 |
| security_group | 10 |
| security_group_rule | 100 |
| subnet | 10 |
| subnetpool | -1 |
+---------------------+-------+

The comment #1 is actually a snippet from the Neutron DB: looks like there is a race condition and the usage value is not recalculated after all floating IPs have been deleted.

[Roman Podoliaka (rpodolyaka)]

754d49398171c9a441087ecc - - -] Attempting to reserve 1 items for resource floatingip. Total usage: 50; quota limit: 50; headroom:0 make_reserv
ation /usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:199
2015-12-18 13:40:30.067 27584 INFO neutron.api.v2.resource [req-4057cd87-cb1d-474d-bd81-8312a3f59b85 69c3f3f211d1424bb060296689698d26 7ff88ebe7
54d49398171c9a441087ecc - - -] create failed (client error): Quota exceeded for resources: ['floatingip']

[Roman Podoliaka (rpodolyaka)]

Looks like it's not even a race condition, as it can be triggered easily:

create 50 IPs to reach the quota usage value:

for i in $(seq 1 50); do neutron floatingip-create admin_floating_net; done

delete all floating IPs:

for ip in $(neutron floatingip-list -f csv -c id --quote=none | grep -v id); do neutron floatingip-delete $ip; done

try to create one IP:

neutron floatingip-create admin_floating_net

[Marcus Klein (marcus-klein)]

I am encountering the same issue with OpenStack from Ubuntu cloud-archives.

mysql> SELECT q.tenant_id,q.in_use,COUNT(f.floating_ip_address) AS floating_ips FROM quotausages q LEFT JOIN floatingips f USING(tenant_id) WHERE q.resource='floatingip' GROUP BY q.tenant_id;
+----------------------------------+--------+--------------+
| tenant_id | in_use | floating_ips |
+----------------------------------+--------+--------------+
| 08609b63401f43679af078d4c22f3f08 | 4 | 4 |
| 3a1e97ffa74f429b8c275fe2ff90c1db | 2 | 2 |
| 7829521236b143d2a6778e09ba588ec0 | 10 | 0 |
| 8d04de0a40cc4419aed0d33a8d5e8eee | 19 | 19 |
| d7400e965d5a40a79708649ae64a9d85 | 1 | 0 |
| f62d544271e6416ba1d334e26133461d | 5 | 5 |
+----------------------------------+--------+--------------+
6 rows in set (0.00 sec)

How can this be solved temporarily? Is it okay to set the in_use value to the correct value?

[Marcus Klein (marcus-klein)]

I will use now the following workaround to fix wrong quota usages for floating ips:

UPDATE quotausages q SET q.in_use = (SELECT COUNT(f.floating_ip_address) FROM floatingips f WHERE f.tenant_id=q.tenant_id) WHERE q.resource='floatingip' AND q.in_use != (SELECT COUNT(f.floating_ip_address) FROM floatingips f WHERE f.tenant_id=q.tenant_id);

A cron job fixes this now every 30 minutes in my installation.

[Sergey Belous (sbelous)]

Fix is on review in upstream: https://review.openstack.org/#/c/269050/

[Alexander Ignatov (aignatov)]

Update: Fix is still in review, struggling with unit tests.

[Oleg Bondarev (obondarev)]

https://review.openstack.org/#/c/274320 is backport for liberty

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Oleg Bondarev <email address hidden>
Review: https://review.fuel-infra.org/16602

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/16602
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: f82049048bb2eb959000f531ad8bd91dd2c3ed24
Author: Oleg Bondarev <email address hidden>
Date: Tue Feb 2 07:55:16 2016

Merge the tip of origin/stable/liberty into origin/openstack-ci/fuel-8.0/liberty

Note: commit ed7ad25 Revert "Revert "Revert "Remove TEMPEST_CONFIG_DIR in the api tox env"""
sets min tox version to 2.3.1 while we currently use 1.9.2.
This patch sets it back to 1.9.2 in order for tests to pass.
Will be reverted back to 2.3.1 once https://bugs.launchpad.net/fuel/+bug/1540516
is fixed.

8476f6f Add relationship between port and floating ip
0bd401c DVR: optimize check_ports_exist_on_l3_agent()
9246cff Change check_ports_exist_on_l3agent to pass the subnet_ids
a133de3 Keep reading stdout/stderr until after kill
ed7ad25 Revert "Revert "Revert "Remove TEMPEST_CONFIG_DIR in the api tox env"""
05f8099 Ensure that tunnels are fully reset on ovs restart
b908c55 Update HA router state if agent is not active
aebd27f Resync L3, DHCP and OVS/LB agents upon revival
8e685c8 Fix floatingip status for an HA router
80c9e84 DVR:Fix _notify_l3_agent_new_port for proper arp update
c12bf81 Fix L3 HA with IPv6
2298566 Make object creation methods in l3_hamode_db atomic
0cc889f Cache the ARP entries in L3 Agent for DVR
8bde9c4 Cleanup veth-pairs in default netns for functional tests
2b96f42 Do not prohibit VXLAN over IPv6
1ab1e58 Fix get_subnet_for_dvr() to return correct gateway mac
3b42dee Imported Translations from Zanata
ca193d0 Revert "Change function call order in ovs_neutron_agent."
96d4ab3 Remove check on dhcp enabled subnets while scheduling dvr
f5299d3 Check gateway ip when update subnet
0d5d7c7 Add tests that constrain db query count
8fb3f9d Don't call add_ha_port inside a transaction
a370fa3 Log INFO message when setting admin state up flag to False for OVS port
bf92dbb DVR: notify specific agent when deleting floating ip
99d1c0d Call _allocate_vr_id outside of transaction
2468b3d Move notifications before DB retry decorator
1b609d2 DVR: handle dvr serviceable port's host change
ad75ccc Imported Translations from Zanata
2e6e135 Run functional gate jobs in a constrained environment
6902c87 DVR: notify specific agent when creating floating ip
00b800d Tox: Remove fullstack env, keep only dsvm-fullstack
d11e9cb Force L3 agent to resync router it could not configure
42f4332 Support migrating of legacy routers to HA and back
4d85fa1 Updated from global requirements
f175cd7 ML2: Add tests to validate quota usage tracking
79d4a08 test_migrations: Avoid returning a filter object for python3
745b546 Do not autoreschedule routers if l3 agent is back online
430892a Avoid full_sync in l3_agent for router updates
fa9fba2 In port_dead, handle case when port already deleted
1d8aff3 Add compatibility with iproute2 >= 4.0

Conflicts:
 neutron/db/l3_dvr_db.py
 neutron/db/l3_hamode_db.py
 neutron/tests/functional/services/l3_router/test_l3_dvr_router_plugin.py
 neutron/tests/unit/agent/l3/test_agent.py

Closes-Bug: #1496341
Closes-Bug: #1531244
Closes-Bug: #1527581
Closes-Bug: #1528201
Closes-Bug: #1528207
Closes...

[Roman Podoliaka (rpodolyaka)]

Fix merged ^

[Oleg Bondarev (obondarev)]

Marking as invalid for 9.0 as fix is in Mitaka already

[Kristina Berezovskaia (kkuznetsova)]

Verify on:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "521"
  build_id: "521"
  fuel-nailgun_sha: "bae6d0062e0825d81409a04bcb4979302f8c65ea"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "29829b131ca802830bc5a9a131c83cd0f43f702b"
  fuel-ostf_sha: "7bcddf18020f2d94a553a441ff57dff9632865df"
  fuel-mirror_sha: "c25d8931e30322ecf43246c8a957e376259b685c"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"
(neutron+vxlan)

Run this script several times:

neutron quota-update --floatingip 5
for i in $(seq 1 5); do neutron floatingip-create admin_floating_net; done
neutron floatingip-create admin_floating_net
for ip in $(neutron floatingip-list -f csv -c id --quote=none | grep -v id); do neutron floatingip-delete $ip; done
neutron floatingip-create admin_floating_net

On iso #478 error appeared on second iteration, on 521 env no error after several times.

[Ekaterina Shutova (eshutova)]

Test case is automated: https://review.gerrithub.io/#/c/298131/