Comment 10 for bug 851159

On Sat, 11 Feb 2012, Lionel Porcheron wrote:

> This week we had an oportunity to see this bug on our internal cloud:
> We rebooted the whole cloud for maintenance purposes (200 vm hosted on 10 servers with a cloud controller)
> When servers are booting, the API node is such overloaded by cloud-init waiting meta-data that is was mostly unusable. We were unable to make more than 1 API call per 2 to 5 min during several hours!
>
> We have also tried to do some tunning on the pool size of sqlachemy
> without a lot of success. The only solution was to drop the DNAT rules
> for redirecting the meta-data calls to makes things usable again.
>
> For the record, the cloud controler is a Xeon Quad Core 2.5 Ghz with 8
> GB of RAM. Now slow query have been seen on MySQL, so it's a pure python
> code that was taking time (and CPU, nova-api was taking all the time
> 100% of CPU).

Were you using something recent? This is known to be terribly broken in
diablo. However it should be at least better in essex.

Additionally, I've tested the HP public cloud, and their metadata service
returns from "python -c 'import boto.utils;
boto.utils.get_instance_metadata()' in < .2 seconds.

I think what they've done is put something caching in the middle as the
first request I did just now took maybe 4 seconds, but the second and
subsequent took < .2 seconds.