For concurrently deployments from 31 to 36  Rally distributes environment among 10 tenants, the maximum number of environments created in one tenant was only 7.

Bug was reproduced on test with 32 concurrently deployments. The env with error was deployed in tenant with another 5 environments.

As seen from the logs for two environments which deployed by different murano-engines there is a try to create interface for murano-default-router using net with subnet with the same cidr, what leads to the error for one of them:

============Murano-engine sent template for first env:
/var/log/murano-all.log.4.gz:<134>Oct  7 03:42:59 node-18 murano-engine Pushing: {'heat_template_version': '2013-05-23', 'description': 'This stack was generated by Murano for environment rally_JkRpl7IA4t (ID: f4830774694b47219a30bdee37bab950)', 'resources': {u'MuranoSecurityGroup-rally_JkRpl7IA4t': {'type': 'OS::Neutron::SecurityGroup', 'properties': {'rules': [{'port_range_min': None, 'port_range_max': None, 'protocol': 'icmp', 'remote_ip_prefix': '0.0.0.0/0'}, {'protocol': u'tcp', 'port_range_max': 25, 'port_range_min': 25, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 53, 'port_range_min': 53, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 53, 'port_range_min': 53, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 88, 'port_range_min': 88, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 88, 'port_range_min': 88, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 123, 'port_range_min': 123, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 135, 'port_range_min': 135, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 137, 'port_range_min': 137, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 138, 'port_range_min': 138, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 445, 'port_range_min': 445, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 445, 'port_range_min': 445, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 464, 'port_range_min': 464, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 464, 'port_range_min': 464, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 389, 'port_range_min': 389, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 389, 'port_range_min': 389, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 636, 'port_range_min': 636, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3268, 'port_range_min': 3268, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3269, 'port_range_min': 3269, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 5722, 'port_range_min': 5722, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 9389, 'port_range_min': 9389, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 65535, 'port_range_min': 49152, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 65535, 'port_range_min': 49152, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3389, 'port_range_min': 3389, 'remote_ip_prefix': '0.0.0.0/0'}], 'description': 'Composite security group of Murano environment rally_JkRpl7IA4t'}}, 'ri-faed2937ab764b898aa8c7ddbbb797dc': {'type': 'OS::Neutron::RouterInterface', 'properties': {'router_id': u'e362a880-0a6c-4709-908d-2819a0fb5ed0', 'subnet': {'get_resource': 'subnet-faed2937ab764b898aa8c7ddbbb797dc'}}}, 'subnet-faed2937ab764b898aa8c7ddbbb797dc': {'type': 'OS::Neutron::Subnet', 'properties': {'ip_version': 4, 'cidr': u'10.0.10.0/24', 'dns_nameservers': [u'8.8.8.8'], 'network': {'get_resource': 'network-faed2937ab764b898aa8c7ddbbb797dc'}}}, 'network-faed2937ab764b898aa8c7ddbbb797dc': {'type': 'OS::Neutron::Net', 'properties': {'name': 'rally_JkRpl7IA4t-network-faed2937ab764b898aa8c7ddbbb797dc'}}}}

================Murano-engine sent template for second env:
/var/log/murano-all.log.3.gz:<134>Oct  7 03:43:03 node-1 murano-engine Pushing: {'heat_template_version': '2013-05-23', 'description': 'This stack was generated by Murano for environment rally_RyKv16zJVz (ID: 5e61b62059c24df9925eb83ea3b4eb09)', 'resources': {'network-5a88edfa8b0a45048933e10fb1a72ca4': {'type': 'OS::Neutron::Net', 'properties': {'name': 'rally_RyKv16zJVz-network-5a88edfa8b0a45048933e10fb1a72ca4'}}, u'MuranoSecurityGroup-rally_RyKv16zJVz': {'type': 'OS::Neutron::SecurityGroup', 'properties': {'rules': [{'port_range_min': None, 'port_range_max': None, 'protocol': 'icmp', 'remote_ip_prefix': '0.0.0.0/0'}, {'protocol': u'tcp', 'port_range_max': 25, 'port_range_min': 25, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 53, 'port_range_min': 53, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 53, 'port_range_min': 53, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 88, 'port_range_min': 88, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 88, 'port_range_min': 88, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 123, 'port_range_min': 123, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 135, 'port_range_min': 135, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 137, 'port_range_min': 137, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 138, 'port_range_min': 138, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 445, 'port_range_min': 445, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 445, 'port_range_min': 445, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 464, 'port_range_min': 464, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 464, 'port_range_min': 464, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 389, 'port_range_min': 389, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 389, 'port_range_min': 389, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 636, 'port_range_min': 636, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3268, 'port_range_min': 3268, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3269, 'port_range_min': 3269, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 5722, 'port_range_min': 5722, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 9389, 'port_range_min': 9389, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 65535, 'port_range_min': 49152, 'remote_mode': 'remote_group_id'}, {'protocol': u'udp', 'port_range_max': 65535, 'port_range_min': 49152, 'remote_mode': 'remote_group_id'}, {'protocol': u'tcp', 'port_range_max': 3389, 'port_range_min': 3389, 'remote_ip_prefix': '0.0.0.0/0'}], 'description': 'Composite security group of Murano environment rally_RyKv16zJVz'}}, 'subnet-5a88edfa8b0a45048933e10fb1a72ca4': {'type': 'OS::Neutron::Subnet', 'properties': {'ip_version': 4, 'cidr': u'10.0.10.0/24', 'dns_nameservers': [u'8.8.8.8'], 'network': {'get_resource': 'network-5a88edfa8b0a45048933e10fb1a72ca4'}}}, 'ri-5a88edfa8b0a45048933e10fb1a72ca4': {'type': 'OS::Neutron::RouterInterface', 'properties': {'router_id': u'e362a880-0a6c-4709-908d-2819a0fb5ed0', 'subnet': {'get_resource': 'subnet-5a88edfa8b0a45048933e10fb1a72ca4'}}}}}

==============Second env gets error:
<135>Oct  7 03:50:25 node-18 murano-api RabbitMessage.Init: message {u'_unique_id': u'a80e4ca3e463494989e87d27a41b29bb', u'_msg_id': u'ac6d52ae79ff4c3293bf3e07d879165f', u'args': {u'environment_id': u'5e61b62059c24df9925eb83ea3b4eb09', u'result': {u'action': {u'isException': True, u'result': {u'message': u'[exceptions.EnvironmentError]: Unexpected stack state UPDATE_FAILED: BadRequest: resources.ri-5a88edfa8b0a45048933e10fb1a72ca4: Bad router request: Cidr 10.0.10.0/24 of subnet 473e90de-8447-4e6b-bfe0-55b8216a1039 overlaps with cidr 10.0.10.0/24 of subnet 6479e12e-5480-4885-a553-ed0d326ecdd7', u'details': u'exceptions.EnvironmentError: Unexpected stack state UPDATE_FAILED: BadRequest: resources.ri-5a88edfa8b0a45048933e10fb1a72ca4: Bad router request: Cidr 10.0.10.0/24 of subnet 473e90de-8447-4e6b-bfe0-55b8216a1039 overlaps with cidr 10.0.10.0/24 of subnet 6479e12e-5480-4885-a553-ed0d326ecdd7\nTraceback (most recent call last):\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano/Classes/Environment.yaml", line 74:9 in method deploy of class io.murano.Environment\n      $.applications.pselect($.deploy())\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano.apps.activeDirectory.ActiveDirectory/Classes/ActiveDirectory.yaml", line 56:9 in method deploy of class io.murano.apps.activeDirectory.ActiveDirectory\n      $.primaryController.deploy()\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano.apps.activeDirectory.ActiveDirectory/Classes/PrimaryController.yaml", line 38:9 in method deploy of class io.murano.apps.activeDirectory.PrimaryController\n      $.super($.deploy())\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano.apps.activeDirectory.ActiveDirectory/Classes/Controller.yaml", line 28:11 in method deploy of class io.murano.apps.activeDirectory.Controller\n      $.host.deploy()\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano.apps.activeDirectory.ActiveDirectory/Classes/Host.yaml", line 38:9 in method deploy of class io.murano.apps.activeDirectory.Host\n      $.super($.deploy())\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano/Classes/resources/Instance.yaml", line 89:9 in method deploy of class io.murano.resources.Instance\n      $.ensureNetworksDeployed()\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano/Classes/resources/Instance.yaml", line 159:13 in method ensureNetworksDeployed of class io.murano.resources.Instance\n      $.environment.defaultNetworks.environment.deploy()\n  File "/tmp/murano-packages-cache/4a40269d-edc1-4b13-81ae-244ab42b812c/io.murano/Classes/resources/NeutronNetwork.yaml", line 67:13 in method deploy of class io.murano.resources.NeutronNetwork\n      $._environment.stack.push()\n  File "/usr/lib/python2.7/dist-packages/murano/engine/system/heat_stack.py", line 193 in method push\n      lambda status: status == \'UPDATE_COMPLETE\')\n  File "/usr/lib/python2.7/dist-packages/murano/engine/system/heat_stack.py", line 143 in method _wait_state\n      "Unexpected stack state {0}{1}".format(status, reason))'}}, u'model': {u'ObjectsCopy': {u'defaultNetworks': {u'environment': {u'useDefaultDns': True, u'name': u'rally_RyKv16zJVz-network', u'dnsNameserver': u'8.8.8.8', u'externalRouterId': u'e362a880-0a6c-4709-908d-2819a0fb5ed0', u'autogenerateSubnet': True, u'subnetCidr': u'10.0.10.0/24', u'autoUplink': True, u'?': {u'type': u'io.murano.resources.NeutronNetwork', u'id': u'5a88edfa8b0a45048933e10fb1a72ca4'}}, u'flat': None}, u'name': u'rally_RyKv16zJVz', u'?': {u'type': u'io.murano.Environment', u'id': u'5e61b62059c24df9925eb83ea3b4eb09'}, u'applications': [{u'name': u'my.domain-LTC7D5AWtGWVWNen', u'adminPassword': u'P@ssw0rd', u'adminAccountName': u'Administrator', u'primaryController': {u'host': {u'availabilityZone': u'nova', u'name': u'murano-1', u'securityGroupName': None, u'assignFloatingIp': False, u'adminPassword': u'asr5znW0tq!', u'ipAddresses': [], u'networks': {u'useFlatNetwork': False, u'primaryNetwork': None, u'useEnvironmentNetwork': True, u'customNetworks': []}, u'keyname': None, u'sharedIps': [], u'floatingIpAddress': None, u'adminAccountName': u'Administrator', u'flavor': u'm1.medium', u'image': u'Murano_windows_image', u'?': {u'type': u'io.murano.apps.activeDirectory.Host', u'id': u'73d3827f-b15b-463e-8012-de3d15f54076'}}, u'recoveryPassword': u'P@ssw0rd', u'?': {u'type': u'io.murano.apps.activeDirectory.PrimaryController', u'id': u'e67b587a-64a7-4a77-9849-345c916b1f62'}, u'dnsIp': None}, u'secondaryControllers': [], u'?': {u'type': u'io.murano.apps.activeDirectory.ActiveDirectory', u'id': u'2ac09a3b-aae2-4530-9e35-2bca474feb0b'}}]}, u'Attributes': [[u'5e61b62059c24df9925eb83ea3b4eb09', u'io.murano.Environment', u'generatedEnvironmentName', u'fmgesifg8yqty8w']], u'Objects': {u'defaultNetworks': {u'environment': {u'useDefaultDns': True, u'name': u'rally_RyKv16zJVz-network', u'dnsNameserver': u'8.8.8.8', u'externalRouterId': u'e362a880-0a6c-4709-908d-2819a0fb5ed0', u'autogenerateSubnet': True, u'subnetCidr': u'10.0.10.0/24', u'autoUplink': True, u'?': {u'_actions': {}, u'type': u'io.murano.resources.NeutronNetwork', u'id': u'5a88edfa8b0a45048933e10fb1a72ca4'}}, u'flat': None}, u'name': u'rally_RyKv16zJVz', u'?': {u'_actions': {u'5e61b62059c24df9925eb83ea3b4eb09_deploy': {u'enabled': True, u'name': u'deploy'}}, u'type': u'io.murano.Environment', u'id': u'5e61b62059c24df9925eb83ea3b4eb09'}, u'applications': [{u'name': u'my.domain-LTC7D5AWtGWVWNen', u'adminPassword': u'P@ssw0rd', u'adminAccountName': u'Administrator', u'primaryController': {u'host': {u'availabilityZone': u'nova', u'name': u'murano-1', u'securityGroupName': None, u'assignFloatingIp': False, u'adminPassword': u'asr5znW0tq!', u'ipAddresses': [], u'networks': {u'useFlatNetwork': False, u'primaryNetwork': None, u'useEnvironmentNetwork': True, u'customNetworks': []}, u'keyname': None, u'sharedIps': [], u'floatingIpAddress': None, u'adminAccountName': u'Administrator', u'flavor': u'm1.medium', u'image': u'Murano_windows_image', u'?': {u'_actions': {}, u'type': u'io.murano.apps.activeDirectory.Host', u'id': u'73d3827f-b15b-463e-8012-de3d15f54076'}}, u'recoveryPassword': u'P@ssw0rd', u'?': {u'_actions': {}, u'type': u'io.murano.apps.activeDirectory.PrimaryController', u'id': u'e67b587a-64a7-4a77-9849-345c916b1f62'}, u'dnsIp': None}, u'secondaryControllers': [], u'?': {u'_actions': {}, u'type': u'io.murano.apps.activeDirectory.ActiveDirectory', u'id': u'2ac09a3b-aae2-4530-9e35-2bca474feb0b'}}]}, u'SystemData': {}}}}, u'method': u'process_result', u'_reply_q': u'reply_9220ba030bbb46faaa44127e690a3ed6'}


So it looks like concurrency issue: probably the second engine doesn't get such cidr in neutron response on 'subnet' list as the subnet with such cidr hasn't been created at the time of request processing.

BTW here is the example of heat template for env http://paste.openstack.org/show/475759/