Cinder

Excessive queries for API volumes/detail extensions

Bug #1197612 reported by Kieran Spear
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
High
Luis A. Garcia
Cinder 2014.1 "icehouse"
Havana
Fix Released
High
Jay Bryant
Cinder 2013.2.1

Bug Description

There are three volumes detail extensions that I can see that are using a separate loop over the list of volumes and making a DB call for each one:

api/contrib/volume_image_metadata.py

api/contrib/volume_host_attribute.py
api/contrib/volume_tenant_attribute.py

The latter two are admin-only by default and are querying for attributes that were present in the original volumes controller query. The first one is getting info from a different table, but should probably be doing it in a single query.

I think Nova solves this issue by caching the objects from the core controller and making them available to extensions, maybe something similar should be done here?

See original description
Tags: api
Kieran Spear (kspear)
description: updated
Mike Perez (thingee)
tags: added: api
Haomai Wang (haomai)
Changed in cinder:
status: New → Confirmed
Luis A. Garcia (luisg-8)
Changed in cinder:
assignee: nobody → Luis A. Garcia (luisg-8)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/55326

Changed in cinder:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/55327

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/55328

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/55329

Jay Bryant (jsbryant)
tags: added: havana-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/55326
Committed: http://github.com/openstack/cinder/commit/bb2daca2864a5607635a3aa37f3874fdda22797b
Submitter: Jenkins
Branch: master

commit bb2daca2864a5607635a3aa37f3874fdda22797b
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 02:19:02 2013 +0000

    Add call to retrieve image metadata for volumes in bulk

    When using the GET volume details REST API call, the image metadata API
    contribution is making an individual db call for each of the available
    volumes. When the number of volumes is large the details call can take
    several minutes.

    This patch adds a call to the volume.API to retrieve image metadata in
    bulk, very similar to the one used to retrieve individual volume image
    metadata.

    Change-Id: Ic3aa721016704c72b7564cc5ceff71676806a24a
    Partial-Bug: #1197612

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/55327
Committed: http://github.com/openstack/cinder/commit/56e5805203a14acad78dc806247b5f0f4a22c9f8
Submitter: Jenkins
Branch: master

commit 56e5805203a14acad78dc806247b5f0f4a22c9f8
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 02:22:30 2013 +0000

    Retrieve volume image metadata using single query

    The image metadata REST API contributions are making an individual db
    call for each of the available volumes. When the number of volumes is
    large the volume details call can take several minutes.

    This patch changes the image metadata API contributions to take
    advantage of the new db query to retrieve metadata in bulk.

    Change-Id: I9a35438c1f38ea8a3d8f5b687ae58ba1f3f78121
    Partial-Bug: #1197612

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/55328
Committed: http://github.com/openstack/cinder/commit/233430b51ddbd207f22eee98386ffbb766fd777f
Submitter: Jenkins
Branch: master

commit 233430b51ddbd207f22eee98386ffbb766fd777f
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 19:37:27 2013 +0000

    Enable object caching in cinder REST API requests

    Allow the core API to cache resources, such as DB results, so that
    extensions can use data already retrieved within the same API request
    eliminating additional expensive DB calls.

    Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
    nova.

    Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
    Partial-Bug: #1197612

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/55329
Committed: http://github.com/openstack/cinder/commit/9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11
Submitter: Jenkins
Branch: master

commit 9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 20:41:02 2013 +0000

    Use cached volumes in REST API extensions

    This patch caches db volumes in the core volumes API so that they can be
    used in the host and tenant attribute API extensions in order to lower
    the response time by avoiding costly individual db queries for each
    volume in the REST API contributions.

    Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
    Partial-Bug: #1197612

Luis A. Garcia (luisg-8)
Changed in cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/56967

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/56968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/56969

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/56970

Thierry Carrez (ttx)
Changed in cinder:
milestone: none → icehouse-1
status: Fix Committed → Fix Released
John Griffith (john-griffith)
Changed in cinder:
importance: Undecided → High
milestone: icehouse-1 → 2013.2.1
milestone: 2013.2.1 → icehouse-1
Alan Pevec (apevec)
tags: removed: havana-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/havana)

Reviewed: https://review.openstack.org/56967
Committed: http://github.com/openstack/cinder/commit/898e8c528cedf53156da9e6ddc8876cf6587d60f
Submitter: Jenkins
Branch: stable/havana

commit 898e8c528cedf53156da9e6ddc8876cf6587d60f
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 02:19:02 2013 +0000

    Add call to retrieve image metadata for volumes in bulk

    When using the GET volume details REST API call, the image metadata API
    contribution is making an individual db call for each of the available
    volumes. When the number of volumes is large the details call can take
    several minutes.

    This patch adds a call to the volume.API to retrieve image metadata in
    bulk, very similar to the one used to retrieve individual volume image
    metadata.

    Change-Id: Ic3aa721016704c72b7564cc5ceff71676806a24a
    Partial-Bug: #1197612
    (cherry picked from commit bb2daca2864a5607635a3aa37f3874fdda22797b)

tags: added: in-stable-havana
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/56968
Committed: http://github.com/openstack/cinder/commit/dbc119b3ec8eff79de2c57ffd602d360040e49e7
Submitter: Jenkins
Branch: stable/havana

commit dbc119b3ec8eff79de2c57ffd602d360040e49e7
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 02:22:30 2013 +0000

    Retrieve volume image metadata using single query

    The image metadata REST API contributions are making an individual db
    call for each of the available volumes. When the number of volumes is
    large the volume details call can take several minutes.

    This patch changes the image metadata API contributions to take
    advantage of the new db query to retrieve metadata in bulk.

    Change-Id: I9a35438c1f38ea8a3d8f5b687ae58ba1f3f78121
    Partial-Bug: #1197612
    (cherry picked from commit 56e5805203a14acad78dc806247b5f0f4a22c9f8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/56969
Committed: http://github.com/openstack/cinder/commit/7f4f083a48e97ff37192caf5bb292c45b0788c92
Submitter: Jenkins
Branch: stable/havana

commit 7f4f083a48e97ff37192caf5bb292c45b0788c92
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 19:37:27 2013 +0000

    Enable object caching in cinder REST API requests

    Allow the core API to cache resources, such as DB results, so that
    extensions can use data already retrieved within the same API request
    eliminating additional expensive DB calls.

    Loosely based on commit 9f9fbc54e7336da10fc3056bdaca2ec7d01c7f94 from
    nova.

    Change-Id: If9f49faf7305287c0489ad6209cf19b8bec612cc
    Partial-Bug: #1197612
    (cherry picked from commit 233430b51ddbd207f22eee98386ffbb766fd777f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/56970
Committed: http://github.com/openstack/cinder/commit/112604485a47a8b83a4036e9797e2fd818be69f5
Submitter: Jenkins
Branch: stable/havana

commit 112604485a47a8b83a4036e9797e2fd818be69f5
Author: Luis A. Garcia <email address hidden>
Date: Tue Nov 5 20:41:02 2013 +0000

    Use cached volumes in REST API extensions

    This patch caches db volumes in the core volumes API so that they can be
    used in the host and tenant attribute API extensions in order to lower
    the response time by avoiding costly individual db queries for each
    volume in the REST API contributions.

    Change-Id: Ie0cbe929379df32fddbdce1953f13b6f26208bff
    Partial-Bug: #1197612
    (cherry picked from commit 9ed0f39fceaaa7a56e74055b54b7f6124e9e7f11)

Alan Pevec (apevec)
tags: removed: in-stable-havana
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/61361

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/61366

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/62531

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/62532

Thierry Carrez (ttx)
Changed in cinder:
milestone: icehouse-1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.