euca-describe-instances is not optimized and takes 38 seconds for 700 VMs

Bug #1157437 reported by Joe Gordon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Joe Gordon

Bug Description

using Devstack (Grizzly-3) and nova SHA1 98f9f735d7e6ac32a8736971f4aab0d81c56131e

and running as admin user.

euca-describe-instances (POST /services/Cloud/ CloudController:DescribeInstances)  is 13 times slower than nova list with 700 VMS.

This is because s3_image_get_by_uuid is called three times per instance and

block_device_mapping_get_all_by_instance,
block_device_mapping_get_all_by_instance,
aggregate_metadata_get_by_host,
get_ec2_instance_id_by_uuid

are called once per hose.

So for 700 VMs that is 4900 DB calls!

In comparison 'nova list' does 2 DB calls no matter how many VMs there are.

When running the same thing as demo user I get the same thing

Joe Gordon (jogo)
description: updated
Joe Gordon (jogo)
description: updated
Revision history for this message
Joe Gordon (jogo) wrote :

Many of these db calls can be removed by using nova.openstack.common.memorycache via the same pattern as in nova/api/metadata/handler.py

tags: added: grizzly-rc-potential
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/24863

Changed in nova:
assignee: nobody → Vish Ishaya (vishvananda)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/25042

Changed in nova:
assignee: Vish Ishaya (vishvananda) → Joe Gordon (jogo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/24863
Committed: http://github.com/openstack/nova/commit/76844433f69a7c29ed4566ad34d7e9740feaf660
Submitter: Jenkins
Branch: master

commit 76844433f69a7c29ed4566ad34d7e9740feaf660
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Mar 19 17:56:30 2013 -0700

    Add caching for ec2 mapping ids.

    Requests to map ec2 ids can be slow when there are a large number
    of instances. These shouldn't ever change so cache them in memory
    for a week. The one week timeout is simply to prevent memory from
    growing uncontrollably.

    Fixes bug 1157437

    Change-Id: I3f1c68959a933a7aaa50f122c34a716264c8722d

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
importance: High → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/25042
Committed: http://github.com/openstack/nova/commit/7ab98e935a63454d0736e6f8a7525a8e73cf6c04
Submitter: Jenkins
Branch: master

commit 7ab98e935a63454d0736e6f8a7525a8e73cf6c04
Author: Joe Gordon <email address hidden>
Date: Thu Mar 21 18:55:49 2013 +0000

    Remove unnecessary DB call to find EC2 AZs.

    Update get_availability_zone_by_host, so it doesn't need a list of services,
    and remove unneeded service_get_all_by_host calls. This removes N DB calls in
    the EC2 api when doing a DescribeInstances, where N is number if instances
    that are being described.

    Partially fix bug 1157437

    Change-Id: I0050ba736828faad94549bc6e55f4ca1d05dcbe7

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25399

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25400

Changed in nova:
milestone: none → grizzly-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (milestone-proposed)

Reviewed: https://review.openstack.org/25399
Committed: http://github.com/openstack/nova/commit/e54700528a1cb3ead26eb0f2d1c1d0932885f07e
Submitter: Jenkins
Branch: milestone-proposed

commit e54700528a1cb3ead26eb0f2d1c1d0932885f07e
Author: Vishvananda Ishaya <email address hidden>
Date: Tue Mar 19 17:56:30 2013 -0700

    Add caching for ec2 mapping ids.

    Requests to map ec2 ids can be slow when there are a large number
    of instances. These shouldn't ever change so cache them in memory
    for a week. The one week timeout is simply to prevent memory from
    growing uncontrollably.

    Fixes bug 1157437

    Change-Id: I3f1c68959a933a7aaa50f122c34a716264c8722d
    (cherry picked from commit 76844433f69a7c29ed4566ad34d7e9740feaf660)

Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/25400
Committed: http://github.com/openstack/nova/commit/c8cb579663fb0464f4c005914567cf8f223f075f
Submitter: Jenkins
Branch: milestone-proposed

commit c8cb579663fb0464f4c005914567cf8f223f075f
Author: Joe Gordon <email address hidden>
Date: Thu Mar 21 18:55:49 2013 +0000

    Remove unnecessary DB call to find EC2 AZs.

    Update get_availability_zone_by_host, so it doesn't need a list of services,
    and remove unneeded service_get_all_by_host calls. This removes N DB calls in
    the EC2 api when doing a DescribeInstances, where N is number if instances
    that are being described.

    Partially fix bug 1157437

    Change-Id: I0050ba736828faad94549bc6e55f4ca1d05dcbe7
    (cherry picked from commit 7ab98e935a63454d0736e6f8a7525a8e73cf6c04)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.