[Murano][Neutron] Periodically neutron failed to create network

[Victor Ryzhenkin]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "521"
  build_id: "2015-06-08_06-13-27"
  nailgun_sha: "4340d55c19029394cd5610b0e0f56d6cb8cb661b"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "7766818f079881e2dbeedb34e1f67e517ed7d479"
  fuel-library_sha: "f43c2ae1af3b493ee0e7810eab7bb7b50c986c7d"
  fuel-ostf_sha: "7c938648a246e0311d05e2372ff43ef1eb2e2761"
  fuelmain_sha: "bcc909ffc5dd5156ba54cae348b6a07c1b607b24"

Steps:
1. Deploy OS-neutron-VLAN, Murano, Ceilo, Sahara enabled. All Ceph, HA (3 KVM controllers, 1 Hardware compute)
2. Deploy k8s cluster with Murano

Actual result:

In some cases Murano deployment fails with error:

 exceptions.IndexError: list index out of range
Traceback (most recent call last):
  File "/tmp/murano-packages-cache/54e3313d-4206-4885-844d-befc9a5acf07/io.murano/Classes/Environment.yaml", line 66:13 in method deploy of class io.murano.Environment
      $.applications.pselect($.deploy())
  File "/tmp/murano-packages-cache/54e3313d-4206-4885-844d-befc9a5acf07/io.murano.apps.docker.kubernetes.KubernetesCluster/Classes/KubernetesCluster.yaml", line 118:13 in method deploy of class io.murano.apps.docker.kubernetes.KubernetesCluster
      $.masterNode.setupEtcd()
  File "/tmp/murano-packages-cache/54e3313d-4206-4885-844d-befc9a5acf07/io.murano.apps.docker.kubernetes.KubernetesCluster/Classes/KubernetesMasterNode.yaml", line 44:24 in method setupEtcd of class io.murano.apps.docker.kubernetes.KubernetesMasterNode
      $resources.yaml('MasterEtcdSetup.template').bind(dict( name => $.instance.name, ip => $.getIp() ))
  File "/tmp/murano-packages-cache/54e3313d-4206-4885-844d-befc9a5acf07/io.murano.apps.docker.kubernetes.KubernetesCluster/Classes/KubernetesNode.yaml", line 30:15 in method getIp of class io.murano.apps.docker.kubernetes.KubernetesNode
      $.instance.ipAddresses[0]
  File "/usr/lib/python2.7/dist-packages/murano/dsl/macros.py", line 80 in method execute
      helpers.evaluate(self._value, context))
  File "/usr/lib/python2.7/dist-packages/murano/dsl/helpers.py", line 66 in method evaluate
      return execute_instruction(value, func, context)
  File "/usr/lib/python2.7/dist-packages/murano/dsl/helpers.py", line 52 in method execute_instruction
      result = action()
  File "/usr/lib/python2.7/dist-packages/murano/dsl/helpers.py", line 62 in method <lambda>
      func = lambda: evaluate(value.evaluate(context), context, 1)
  File "/usr/lib/python2.7/dist-packages/murano/dsl/yaql_expression.py", line 73 in method evaluate
      return self._parsed_expression.evaluate(context=context)
  File "/usr/lib/python2.7/dist-packages/yaql/expressions.py", line 35 in method evaluate
      return f()
  File "/usr/lib/python2.7/dist-packages/yaql/expressions.py", line 100 in method __call__
      return self._try_invoke(fs, args_to_pass, context)
  File "/usr/lib/python2.7/dist-packages/yaql/expressions.py", line 112 in method _try_invoke
      return func(*args_to_pass)
  File "/usr/lib/python2.7/dist-packages/yaql/functions/builtin.py", line 71 in method get_by_index
      return this[index]

The root cause of this - the Heat can't create resource:

http://paste.openstack.org/show/282230/

And, finally, in neutron-server.log we can find:

http://paste.openstack.org/show/282264/

This issue was reproduced in ~3 of 10 cases.
This issue was not encountered in environments with GRE segmentation.

[Victor Ryzhenkin (vryzhenkin)]

Snapshot: https://drive.google.com/a/mirantis.com/file/d/0B3tqtKJGirwwRU9OZTA5WFl2eHc/view?usp=sharing

[Sergey Kolekonov (skolekonov)]

Do you have enough allocated VLANs for your cluster? This error may mean that Neutron server can't allocate a free VLAN for network

[Ilya Shakhat (shakhat)]

@Victor - does env contain enough VLANs?

[Victor Ryzhenkin (vryzhenkin)]

We have only 2 VLAN's on environment.
Because of this, neutron can't create network.
Moving to invalid.