OK i bumped my mysql connections count to 4K / server, and ran some more tests.
Generally a single heat stack will complete correctly.
If 2 are started @ the same time, it will generally fail.
if 3, all will fail in an exception from neutronclient around subnet creation, as in
logs-16-07-14-13:31:04.tar.gz and snipped below.
OK i bumped my mysql connections count to 4K / server, and ran some more tests.
Generally a single heat stack will complete correctly. 07-14-13: 31:04.tar. gz and snipped below.
If 2 are started @ the same time, it will generally fail.
if 3, all will fail in an exception from neutronclient around subnet creation, as in
logs-16-
- - -] Task create from Subnet "sn1" Stack "t31" [dbd77b02- d5c1-45e0- 96de-7a3d9a954e 1b] running step /usr/local/ lib/python2. 7/dist- packages/ heat/engine/ scheduler. py:226 lib/python2. 7/dist- packages/ neutronclient/ common/ utils.py: 194 fault_response /usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py:247 client [req-020e7186- c09b-4897- 8b5d-faacb75682 b9 a4aab8472ebb41f aa98f89e1dd3485 8a 469300d96f9f406 cacda5f001d387e 52 - - -] RESP: 201 {'Content-Length': '558', 'Strict- Transport- Security' : 'max-age=15768000', 'Server': 'nubo-1', 'Connection': 'keep-alive', 'Date': 'Thu, 14 Jul 2016 13:29:50 GMT', 'Content-Type': 'application/json', 'X-Openstack- Request- Id': 'req-f1986214- 98c0-4690- 9d09-2158763813 73'} {"subnet": {"description": "", "enable_dhcp": true, "network_id": "5542d6e9- 7624-4242- bbd2-7743741c02 1c", "tenant_id": "469300d96f9f40 6cacda5f001d387 e52", "created_at": "2016-07- 14T13:29: 14", "dns_nameservers": [], "updated_at": "2016-07- 14T13:29: 14", "gateway_ip": "172.16.186.1", "ipv6_ra_mode": null, "allocation_pools": [{"start": "172.16.186.9", "end": "172.16.186.254"}], "host_routes": [], "ip_version": 4, "ipv6_address_ mode": null, "cidr": "172.16.186.0/24", "id": "34d3d980- 3429-4054- 8f85-7a8b24d2f3 21", "subnetpool_id": null, "name": "t31_sn186"}} http_log_resp /usr/local/ lib/python2. 7/dist- packages/ neutronclient/ common/ utils.py: 194 resource [req-020e7186- c09b-4897- 8b5d-faacb75682 b9 a4aab8472ebb41f aa98f89e1dd3485 8a 469300d96f9f406 cacda5f001d387e 52 - - -] CREATE: Subnet "sn72" Stack "t31" [dbd77b02- d5c1-45e0- 96de-7a3d9a954e 1b] resource Traceback (most recent call last): resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 732, in _action_recorder resource yield resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 812, in _do_action resource yield self.action_ handler_ task(action, args=handler_args) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ scheduler. py", line 312, in wrapper resource step = next(subtask) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 775, in action_handler_task resource handler_data = handler(*args) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resources/ openstack/ neutron/ subnet. py", line 361, in handle_create resource subnet = self.client( ).create_ subnet( {'subnet' : props})['subnet'] resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 669, in create_subnet resource return self.post( self.subnets_ path, body=body) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 347, in post resource headers=headers, params=params) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 282, in do_request resource self._handle_ fault_response( status_ code, replybody, resp) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 257, in _handle_ fault_response resource exception_ handler_ v20(status_ code, error_body) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 84, in exception_ handler_ v20 resource request_ ids=request_ ids) resource resource lib/python2. 7/dist- packages/ neutronclient/ common/ utils.py: 194 fault_response /usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py:247 resource [req-020e7186- c09b-4897- 8b5d-faacb75682 b9 a4aab8472ebb41f aa98f89e1dd3485 8a 469300d96f9f406 cacda5f001d387e 52 - - -] CREATE: Subnet "sn236" Stack "t31" [dbd77b02- d5c1-45e0- 96de-7a3d9a954e 1b] resource Traceback (most recent call last): resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 732, in _action_recorder resource yield resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 812, in _do_action resource yield self.action_ handler_ task(action, args=handler_args) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ scheduler. py", line 312, in wrapper resource step = next(subtask) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resource. py", line 775, in action_handler_task resource handler_data = handler(*args) resource File "/usr/local/ lib/python2. 7/dist- packages/ heat/engine/ resources/ openstack/ neutron/ subnet. py", line 361, in handle_create resource subnet = self.client( ).create_ subnet( {'subnet' : props})['subnet'] resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 669, in create_subnet resource return self.post( self.subnets_ path, body=body) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 347, in post resource headers=headers, params=params) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 282, in do_request resource self._handle_ fault_response( status_ code, replybody, resp) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 257, in _handle_ fault_response resource exception_ handler_ v20(status_ code, error_body) resource File "/usr/local/ lib/python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 84, in exception_ handler_ v20 resource request_ ids=request_ ids) resource resource
[332B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
http_log_resp /usr/local/
[198B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
_handle_
2016-07-14 13:29:50.827 3982 DEBUG neutronclient.
2016-07-14 13:29:50.828 3988 INFO heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
[87B blob data]
[105B blob data]
[79B blob data]
[103B blob data]
[99B blob data]
[64B blob data]
[64B blob data]
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
2016-07-14 13:29:50.828 3988 ERROR heat.engine.
[332B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
http_log_resp /usr/local/
[198B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]
_handle_
2016-07-14 13:29:50.834 3981 INFO heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
[87B blob data]
[105B blob data]
[79B blob data]
[103B blob data]
[99B blob data]
[64B blob data]
[64B blob data]
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
2016-07-14 13:29:50.834 3981 ERROR heat.engine.
[266B blob data]
[49B blob data]
[23B blob data]
[47B blob data]
[43B blob data]
[8B blob data]
[8B blob data]