OpenStack Compute (Nova)

listing of instances takes progressively longer the more instances there are

Reported by Antony Messerli on 2011-09-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Chris Behrens

Bug Description

Running commit ebec34e374bd9ec5354e3648a6e8f3ce26c951b5, listing of instances gets progressively slower the more instances are in an account. As the number of instances is reduced the nova list becomes more rapid.

Chris Behrens (cbehrens) wrote :

It's got to be the calls to network manager for every single instance. I think there's 2 calls per instance. We can actually cut it down from 2n calls to 1... if we can pass all instances we need IP info for to the network manager in a single rpc.call().

Chris Behrens (cbehrens) wrote :

Looks like it... for what this calls. This issue is more likely the fact that when doing a "nova list", there's now 2 rpc.calls done per instance. This was a recent change, to try to push all network related stuff through the network service with the notion that this information might be stored in an external service. (An external service will only make this worse.) Things were fine before this. Someone from ozone will likely grab this soon and try to cut this down to a single call per 'nova list'.

Note that EC2 likely has a similar problem now as well.

We also could/should consider caching this information.

On Sep 29, 2011, at 7:18 PM, Chris Behrens wrote:

> Looks like it... for what this calls. This issue is more likely the
> fact that when doing a "nova list", there's now 2 rpc.calls done per
> instance. This was a recent change, to try to push all network related
> stuff through the network service with the notion that this information
> might be stored in an external service. (An external service will only
> make this worse.) Things were fine before this. Someone from ozone
> will likely grab this soon and try to cut this down to a single call per
> 'nova list'.
>
> Note that EC2 likely has a similar problem now as well.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/862839
>
> Title:
> listing of instances takes progressively longer the more instances
> there are
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Running commit ebec34e374bd9ec5354e3648a6e8f3ce26c951b5, listing of
> instances gets progressively slower the more instances are in an
> account. As the number of instances is reduced the nova list becomes
> more rapid.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/862839/+subscriptions

Chris Behrens (cbehrens) on 2011-09-30
Changed in nova:
assignee: nobody → Chris Behrens (cbehrens)
status: New → In Progress
Chris Behrens (cbehrens) wrote :

Vish: Agree completely. Reverting some of this stuff for now. Caching is being worked separately. We can't untie this network stuff until we have the caching in place.

https://review.openstack.org/1015

Thierry Carrez (ttx) on 2011-10-21
Changed in nova:
importance: Undecided → Medium

Reviewed: https://review.openstack.org/1123
Committed: http://github.com/openstack/nova/commit/61e5b8e7da3b36db9a09f80d62ebf2e276bbe88b
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 61e5b8e7da3b36db9a09f80d62ebf2e276bbe88b
Author: Chris Behrens <email address hidden>
Date: Fri Oct 21 00:29:54 2011 -0700

    Revert how APIs get IP address info for instances

    Fixes bug 862839

    listing instances with IPs is extremely inefficient after changes were
    made to query the network manager for IP information for each instance.
    I tried adding a network manager call that said 'give me IP information
    for 'x' instances', but that was also too slow.

    We need a solution that caches IP info from the network manager before
    we can fully untie things. So, this reverts APIs to use
    instance['fixed_ips'] which hasn't been untied in the DB yet.

    Change-Id: I37d21105d6306f0a812c5eb0f0717a5094cd17b9

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-11-09
Changed in nova:
milestone: none → essex-1
Thierry Carrez (ttx) on 2011-11-17
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers