I tried some runs with 8 instances on a host with 8 cores. There were several large gaps indicating serialization. The biggest was in the update_available_resource method in nova/virt/libvirt/connection.py. It was taking as much as 30 seconds. Most of that time seemed to be in getting vcpu_used, cpu_info, and disk_available_least.
For the multiple cpu case there is also always a 5 second gap between
IPTablesManager.apply completed with success
and
Running cmd (subprocess): sudo /usr/local/bin/nova-rootwrap /etc/nova/rootwrap.conf tee /sys/class/net/vnet7/brport/hairpin_mode
All in all, the main problem seems to be that the periodic manager processes (green threads?) are doing slow operations and not sufficiently yielding to the "real" work that is trying to launch instances that truly run in parallel native processes. But I don't understand the nova threading/rpc/etc. architecture enough to really say.
I tried some runs with 8 instances on a host with 8 cores. There were several large gaps indicating serialization. The biggest was in the update_ available_ resource method in nova/virt/ libvirt/ connection. py. It was taking as much as 30 seconds. Most of that time seemed to be in getting vcpu_used, cpu_info, and disk_available_ least.
For the multiple cpu case there is also always a 5 second gap between
IPTablesManager .apply completed with success
and
Running cmd (subprocess): sudo /usr/local/ bin/nova- rootwrap /etc/nova/ rootwrap. conf tee /sys/class/ net/vnet7/ brport/ hairpin_ mode
All in all, the main problem seems to be that the periodic manager processes (green threads?) are doing slow operations and not sufficiently yielding to the "real" work that is trying to launch instances that truly run in parallel native processes. But I don't understand the nova threading/rpc/etc. architecture enough to really say.