creating a PTG is taking a long time with a shared L3 policy across tenants

[Kent Wu]

Just finished one round of configuration of 100 tenants & each tenant has 17 L2s/PTGs/PTs through openStack controller. The one I started this afternoon is not good as it ran into the mcast pool too small issue after the openStack re-install. I did a re-install so it has all the latest stuff including Ivar's fix to address the slow subnet allocation issue when there is only one shared L3 policy. The testing result shows that its definitely much better than before however it still took about 4 hrs & 15 minutes to finish this round (compared to 1 hr & 40 minutes when there is a separate L3 for each tenant). Some observations:

1. at the beginning, each PTG creation would only take about 1 ~ 2 secs
2. it gradually increased then when it reached tenant 100 it took about 12 secs to create a PTG......When that happens, neutron-server was taking about 85+% CPU while sqld was using about 13% CPU. Note that there are 4 cores in total so some cores are still available.
3. This is still a big improvement though as before it would take like 50 secs when it reached tenant 70.

So is this kind of improvement good enough or we need to make it even better? I would like to file a bug for it if its still not good enough.