Increasing number of InstancePCIRequests.get_by_instance_uuid RPC calls during compute host auditing

[James Page]

Environment: Ubuntu 14.04/OpenStack Juno Release

The periodic auditing on compute node becomes very RPC call intensive when a large number of instances are running on a cloud; the InstancePCIRequests.get_by_instance_uuid call is made on all instances running on the hypervisor - when this is multiplied across a large number of hypervisors, this impacts back onto the conductor processes as they try to service an increasing amount of RPC calls over time.

[Dan Smith (danms)]

This is a known consequence of some last-minute code that was pulled into juno. We have patches up for improving this flow during instance build, as part of this work:

https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/flavor-from-sysmeta-to-blob,n,z

This should be specifically related to extra calls during the process of building or resizing instances and not constant additional load. If that's not the case, then please provide extra details.

[James Page (james-page)]

Looking at the log data, the number of RPC calls relates exactly to the number of running instances on the hypervisor, and is definately in the context of the period audit.

I'm running a continual 100 instances parallel instance creation load against the cloud.

[James Page (james-page)]

For reference:

Number of hypervisors: 500
Number of conductors: 8 servers, each with 4 cores/8 threads and 16G of memory, each conductor server is configured with 32 worker threads.
Number of running instances: 29,000

[James Page (james-page)]

The proposed change:

https://review.openstack.org/#/c/131321/1

Appears to resolve this issue.