getting quotas from database has severe performance implications

Bug #1075369 reported by Edgar Magana on 2012-11-06
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Critical
Juergen Brendel
Folsom
Critical
dan wendlandt
quantum (Ubuntu)
Undecided
Unassigned
Quantal
Undecided
Unassigned

Bug Description

In quantum when a new network is created and system checks for user's quota, instead of getting a count from DB object, it returns all objects from db and locally counts them all. It has performance implications when the number of objects increases.

Edgar Magana (emagana) on 2012-11-06
Changed in quantum:
assignee: nobody → Edgar Magana (emagana)
pritesh (pritesh) on 2012-11-06
Changed in quantum:
assignee: Edgar Magana (emagana) → pk (pritkoth)
assignee: pk (pritkoth) → nobody
description: updated
Changed in quantum:
assignee: nobody → Juergen Brendel (jbrendel)
dan wendlandt (danwent) on 2012-11-07
Changed in quantum:
milestone: none → grizzly-1
importance: Undecided → High

Fix proposed to branch: master
Review: https://review.openstack.org/15747

Changed in quantum:
status: New → In Progress
dan wendlandt (danwent) on 2012-11-12
Changed in quantum:
importance: High → Critical
Gary Kotton (garyk) on 2012-11-17
tags: added: folsom-backport-potential

Reviewed: https://review.openstack.org/15747
Committed: http://github.com/openstack/quantum/commit/513307fa916c5461bb9a144b74b562b9d34a1c65
Submitter: Jenkins
Branch: master

commit 513307fa916c5461bb9a144b74b562b9d34a1c65
Author: Juergen Brendel <email address hidden>
Date: Thu Nov 8 15:00:11 2012 +1300

    Use DB count to get resource counts.

    Fixes bug 1075369.

    During quota check we used to simply retrieve the entire collection
    of resources from the database, then counting them in Python. This
    patch introduces a specialized _get_collection_count() method, which
    instead take advantage of the DB's built-in count capabilities.

    In order to take advantage of this, plugins can now implement
    get_*_count() methods for their resources. This is used (if present)
    by the quota checking function.

    Patch incorporates review feedback from Dan W, Alex Xu, Zhongyue Luo,
    Edgar Magana, Akihiro Motoki and gongysh.

    Change-Id: I87e2d0294e116e8147fed2ee90c9eb0cf1a54362

Changed in quantum:
status: In Progress → Fix Committed
dan wendlandt (danwent) wrote :

here is ML post showing how bad things get at large scales: http://lists.openstack.org/pipermail/openstack-dev/2012-November/002375.html

summary: - getting quotas from database has performance implications
+ getting quotas from database has severe performance implications
Thierry Carrez (ttx) on 2012-11-21
Changed in quantum:
status: Fix Committed → Fix Released

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16952

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/17000

Reviewed: https://review.openstack.org/16952
Committed: http://github.com/openstack/quantum/commit/ca7fc10dac532fdea5398b394ac3e58b1c3d8791
Submitter: Jenkins
Branch: stable/folsom

commit ca7fc10dac532fdea5398b394ac3e58b1c3d8791
Author: Dan Wendlandt <email address hidden>
Date: Tue Nov 27 13:14:54 2012 -0800

    Use DB count to get resource counts.

    Fixes bug 1075369, and includes bugs 1081259, 1083696 which fixed
    regression in the original version of this patch and bug 1083180
    which improves the documentation of the original patch.

    During quota check we used to simply retrieve the entire collection
    of resources from the database, then counting them in Python. This
    patch introduces a specialized _get_collection_count() method, which
    instead take advantage of the DB's built-in count capabilities.

    In order to take advantage of this, plugins can now implement
    get_*_count() methods for their resources. This is used (if present)
    by the quota checking function.

    Change-Id: Idc347cfacc363f2cac47356296514fe735625f29

Gary Kotton (garyk) on 2012-11-28
tags: added: in-stable-folsom
removed: db folsom-backport-potential
Changed in quantum (Ubuntu):
status: New → Fix Released
Changed in quantum (Ubuntu Quantal):
status: New → Confirmed

Hello Edgar, or anyone else affected,

Accepted quantum into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/quantum/2012.2.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in quantum (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Mark McLoughlin (markmc) on 2013-01-22
tags: removed: in-stable-folsom
Launchpad Janitor (janitor) wrote :
Download full text (3.8 KiB)

This bug was fixed in the package quantum - 2012.2.1-0ubuntu1

---------------
quantum (2012.2.1-0ubuntu1) quantal-proposed; urgency=low

  * Resynchronize with stable/folsom (1e774867) (LP: #1085255):
    - [aeabb42] There are routing problems when the dnsmasq port does not come
      first in the routing table (LP: #1083238)
    - [04aab72] Quantum linux bridge not optimized with libvirt (LP: #1078210)
    - [ca7fc10] getting quotas from database has severe performance implications
      (LP: #1075369)
    - [66605e8] failed to update an external network into non external network
      (LP: #1083387)
    - [c60051a] Quantum test suite leaks memory like a sieve (LP: #1065276)
    - [3179dfc] clear_db() does incomplete db teardown (LP: #1080988)
    - [c1e19d7] Unauthorized command: cat /proc/None/cmdline (LP: #1077651)
    - [af9e076] At times a instance will not receive an IP address from the DHCP
      agent (LP: #1081664)
    - [e0d1a7d] allow multiple floating-ip on single port if they use different
      fixed ips and/or external nets (LP: #1057844)
    - [8471d79] Delete port fails to gateway ip (LP: #1079980)
    - [aca8b4a] fixed_ip allocation which is not included within
      allocation_pools makes error when delete port or re-create port
      (LP: #1077292)
    - [eacc9d3] Mapping same bridge to different phyiscal networks succeed
      (LP: #1067669)
    - [51b4c82] python-quantum: not region aware (LP: #1080793)
    - [6f0a486] delete floatingip should be in one transaction to delete port
      (LP: #1080516)
    - [db6cda7] Remove qpid configuration variables no longer supported
    - [a112840] Allow NVP plugin to use per-tenant quota extension
    - [82b1a55] Quantum service does not restart after reboot (LP: #1073999)
    - [c01a839] There are some cases that L3 API with an invalid parameter
      returns 500. (LP: #1064765)
    - [26b383f] external network can be plugged also as internal network for one
      router (LP: #1053633)
    - [49f649c] There is a lot of cases that API with an invalid parameter
      returns 500. (LP: #1062046)
    - [4546a18] When create subnet, you con set up the value as cidr (the value
      isn't cidr form). (LP: #1067959)
    - [9ba453a] killfilter should handle updated/deleted executables
      (LP: #1073768)
    - [7c8a55c] a port which is not able to delete is made when floatingip
      create fails. (LP: #1064748)
    - [c9b84cf] Linux bridge port update causes exception (LP: #1072713)
    - [cb57932] I can't add interface to router, if there is another port in
      non-shared network of other tenant (LP: #1057558)
    - [574e278] Ryu plugin does not support Security Groups (LP: #1059393)
    - [607f486] tap device added to integration bridge without tag
      (LP: #1064070)
    - [21a0fdf] L3 agent external network flag (LP: #1056720)
    - [5cbaff4] router create with external_gateway_info fails with 500 always.
      (LP: #1064235)
    - [63b81f6] l3 db operations failed in multiple transactions (LP: #1070335)
    - [bff17fb] Ensure that the SqlSoup import is still supported.
    - [e091a29] l3_nat_agent was renamed to l3_agent
    - [9030969] remove default value of 'local_ip' of 10...

Read more...

Changed in quantum (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-04-04
Changed in quantum:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers