listing of instances takes progressively longer the more instances there are

[Antony Messerli]

Running commit ebec34e374bd9ec5354e3648a6e8f3ce26c951b5, listing of instances gets progressively slower the more instances are in an account. As the number of instances is reduced the nova list becomes more rapid.

[Chris Behrens (cbehrens)]

It's got to be the calls to network manager for every single instance. I think there's 2 calls per instance. We can actually cut it down from 2n calls to 1... if we can pass all instances we need IP info for to the network manager in a single rpc.call().

[Jesse Andrews (anotherjesse)]

Are there indexes on the calls being made?

[Chris Behrens (cbehrens)]

Looks like it... for what this calls. This issue is more likely the fact that when doing a "nova list", there's now 2 rpc.calls done per instance. This was a recent change, to try to push all network related stuff through the network service with the notion that this information might be stored in an external service. (An external service will only make this worse.) Things were fine before this. Someone from ozone will likely grab this soon and try to cut this down to a single call per 'nova list'.

Note that EC2 likely has a similar problem now as well.

[Vish Ishaya (vishvananda)]

We also could/should consider caching this information.

On Sep 29, 2011, at 7:18 PM, Chris Behrens wrote:

> Looks like it... for what this calls. This issue is more likely the
> fact that when doing a "nova list", there's now 2 rpc.calls done per
> instance. This was a recent change, to try to push all network related
> stuff through the network service with the notion that this information
> might be stored in an external service. (An external service will only
> make this worse.) Things were fine before this. Someone from ozone
> will likely grab this soon and try to cut this down to a single call per
> 'nova list'.
>
> Note that EC2 likely has a similar problem now as well.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/862839
>
> Title:
> listing of instances takes progressively longer the more instances
> there are
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Running commit ebec34e374bd9ec5354e3648a6e8f3ce26c951b5, listing of
> instances gets progressively slower the more instances are in an
> account. As the number of instances is reduced the nova list becomes
> more rapid.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/862839/+subscriptions

[Chris Behrens (cbehrens)]

Vish: Agree completely. Reverting some of this stuff for now. Caching is being worked separately. We can't untie this network stuff until we have the caching in place.

https://review.openstack.org/1015

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/1123
Committed: http://github.com/openstack/nova/commit/61e5b8e7da3b36db9a09f80d62ebf2e276bbe88b
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 61e5b8e7da3b36db9a09f80d62ebf2e276bbe88b
Author: Chris Behrens <email address hidden>
Date: Fri Oct 21 00:29:54 2011 -0700

    Revert how APIs get IP address info for instances

    Fixes bug 862839

    listing instances with IPs is extremely inefficient after changes were
    made to query the network manager for IP information for each instance.
    I tried adding a network manager call that said 'give me IP information
    for 'x' instances', but that was also too slow.

    We need a solution that caches IP info from the network manager before
    we can fully untie things. So, this reverts APIs to use
    instance['fixed_ips'] which hasn't been untied in the DB yet.

    Change-Id: I37d21105d6306f0a812c5eb0f0717a5094cd17b9