nova messaging queues slow in idle cluster

[Paul Griffin]

Each running nova-network service periodically calls fixed_ip_disassociate_all_by_timeout(). This searches all fixed ips, filtering on multiple columns. With a large number of fixed_ips, each call to this function can take several seconds to complete. With many hosts running nova-network, a cluster with little to no activity can experience prolonged delays in message processing, eventually rendering some or all hosts unresponsive to nova commands (boot instance, etc). The only column referenced in that query that is not represented in any existing index is updated_at; a new index including fixed_ips.updated_at is probably called for.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/122240

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/122240
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e9f3ad139734593537e912fe92b8717db9254acc
Submitter: Jenkins
Branch: master

commit e9f3ad139734593537e912fe92b8717db9254acc
Author: Paul Griffin <email address hidden>
Date: Wed Sep 17 13:04:20 2014 -0400

    db: Add index on fixed_ips updated_at

    Adding sqlalchemy code to add an index to the fixed_ips table on
    the deleted, allocated and updated_at columns, to speed up the
    periodic disassociate_all_by_timeout task

    Closes-Bug: #1363119
    Change-Id: Iee53b5fd31b56bf1e0dddeada42ee9267b156150