OK i bumped my mysql connections count to 4K / server, and ran some more tests.

Generally a single heat stack will complete correctly.
If 2 are started @ the same time, it will generally fail.
if 3, all will fail in an exception from neutronclient around subnet creation, as in
logs-16-07-14-13:31:04.tar.gz and snipped below.


- - -] Task create from Subnet "sn1" Stack "t31" [dbd77b02-d5c1-45e0-96de-7a3d9a954e1b] running step /usr/local/lib/python2.7/dist-packages/heat/engine/scheduler.py:226
[332B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 http_log_resp /usr/local/lib/python2.7/dist-packages/neutronclient/common/utils.py:194
[198B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 _handle_fault_response /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:247
2016-07-14 13:29:50.827 3982 DEBUG neutronclient.client [req-020e7186-c09b-4897-8b5d-faacb75682b9 a4aab8472ebb41faa98f89e1dd34858a 469300d96f9f406cacda5f001d387e52 - - -] RESP: 201 {'Content-Length': '558', 'Strict-Transport-Security': 'max-age=15768000', 'Server': 'nubo-1', 'Connection': 'keep-alive', 'Date': 'Thu, 14 Jul 2016 13:29:50 GMT', 'Content-Type': 'application/json', 'X-Openstack-Request-Id': 'req-f1986214-98c0-4690-9d09-215876381373'} {"subnet": {"description": "", "enable_dhcp": true, "network_id": "5542d6e9-7624-4242-bbd2-7743741c021c", "tenant_id": "469300d96f9f406cacda5f001d387e52", "created_at": "2016-07-14T13:29:14", "dns_nameservers": [], "updated_at": "2016-07-14T13:29:14", "gateway_ip": "172.16.186.1", "ipv6_ra_mode": null, "allocation_pools": [{"start": "172.16.186.9", "end": "172.16.186.254"}], "host_routes": [], "ip_version": 4, "ipv6_address_mode": null, "cidr": "172.16.186.0/24", "id": "34d3d980-3429-4054-8f85-7a8b24d2f321", "subnetpool_id": null, "name": "t31_sn186"}} http_log_resp /usr/local/lib/python2.7/dist-packages/neutronclient/common/utils.py:194
2016-07-14 13:29:50.828 3988 INFO heat.engine.resource [req-020e7186-c09b-4897-8b5d-faacb75682b9 a4aab8472ebb41faa98f89e1dd34858a 469300d96f9f406cacda5f001d387e52 - - -] CREATE: Subnet "sn72" Stack "t31" [dbd77b02-d5c1-45e0-96de-7a3d9a954e1b]
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource Traceback (most recent call last):
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 732, in _action_recorder
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     yield
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 812, in _do_action
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 312, in wrapper
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     step = next(subtask)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 775, in action_handler_task
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     handler_data = handler(*args)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resources/openstack/neutron/subnet.py", line 361, in handle_create
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     subnet = self.client().create_subnet({'subnet': props})['subnet']
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 669, in create_subnet
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     return self.post(self.subnets_path, body=body)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 347, in post
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     headers=headers, params=params)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 257, in _handle_fault_response
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource     request_ids=request_ids)
[87B blob data]
[105B blob data]
[79B blob data]
[103B blob data]
[99B blob data]
[64B blob data]
[64B blob data]
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource 
2016-07-14 13:29:50.828 3988 ERROR heat.engine.resource 
[332B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 http_log_resp /usr/local/lib/python2.7/dist-packages/neutronclient/common/utils.py:194
[198B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
 _handle_fault_response /usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:247
2016-07-14 13:29:50.834 3981 INFO heat.engine.resource [req-020e7186-c09b-4897-8b5d-faacb75682b9 a4aab8472ebb41faa98f89e1dd34858a 469300d96f9f406cacda5f001d387e52 - - -] CREATE: Subnet "sn236" Stack "t31" [dbd77b02-d5c1-45e0-96de-7a3d9a954e1b]
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource Traceback (most recent call last):
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 732, in _action_recorder
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     yield
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 812, in _do_action
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 312, in wrapper
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     step = next(subtask)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 775, in action_handler_task
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     handler_data = handler(*args)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resources/openstack/neutron/subnet.py", line 361, in handle_create
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     subnet = self.client().create_subnet({'subnet': props})['subnet']
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 669, in create_subnet
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     return self.post(self.subnets_path, body=body)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 347, in post
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     headers=headers, params=params)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 282, in do_request
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     self._handle_fault_response(status_code, replybody, resp)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 257, in _handle_fault_response
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     exception_handler_v20(status_code, error_body)
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource     request_ids=request_ids)
[87B blob data]
[105B blob data]
[79B blob data]
[103B blob data]
[99B blob data]
[64B blob data]
[64B blob data]
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource 
2016-07-14 13:29:50.834 3981 ERROR heat.engine.resource 
[266B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]