High latency on delete_subnets with increasing load and concurrency

[Sridar Kandaswamy]

In our environments on deployments at reasonable scale (100 networks with 4 subnets each) with 3 Controllers and abt 50 Computes we notice high latency on subnet deletion. But even in smaller deployments we are able to see issues using rally create_and_delete_subnets tests[1]. There is a correlation with increasing load and concurrency. These tests were run on the master branch with a devstack based single node deployment(VM). Our actual deployments are on high end physical servers (Bare Metal).

I suspect that issues are around here:
https://github.com/openstack/neutron/blob/master/neutron/db/db_base_plugin_v2.py#L991

I am not sure if this is a known issue and folks already have some thoughts around this.

[1] http://paste.openstack.org/show/749782/