quantum instance list index out of range

[Steve Baker]

From: Pekka Rinne <email address hidden>

I've got a template which gives list index out of range occasionally from heat (grizzly). Any ideas why? sometimes it works ok. Some kind of race situation? I'm just trying to learn quantum usage here..

2013-06-18 18:52:02.585 27948 ERROR heat.engine.resource [-] create Instance "inst1"
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource Traceback (most recent call last):
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat-2013.1.1.a1.g6ed78c2-py2.7.egg/heat/engine/resource.py", line 321, in create
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource while not self.check_active():
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat-2013.1.1.a1.g6ed78c2-py2.7.egg/heat/engine/resources/instance.py", line 325, in check_active
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource self._set_ipaddress(server.networks)
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/heat-2013.1.1.a1.g6ed78c2-py2.7.egg/heat/engine/resources/instance.py", line 125, in _set_ipaddress
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource self.ipaddress = networks[n][0]
2013-06-18 18:52:02.585 27948 TRACE heat.engine.resource IndexError: list index out of range

{
   "AWSTemplateFormatVersion" : "2010-09-09",

   "Description" : "tbd",

   "Parameters" : {

     "KeyName" : {
       "Description" : "Keydescription",
       "Type" : "String"
     },

     "InstanceType" : {
       "Description" : "Instancetype",
       "Type" : "String",
       "Default" : "m1.small",
       "AllowedValues" : [ "m1.small" ],
       "ConstraintDescription" : "must be a valid EC2 instance type."
     },

     "MyDistribution": {
       "Default": "MyDistro",
       "Description" : "Distribution of mine",
       "Type": "String",
       "AllowedValues" : [ "MyDistro" ]
     }

   },

   "Mappings" : {
     "AWSInstanceType2Arch" : {
       "m1.small" : { "Arch" : "64" }
     },
     "DistroArch2AMI": {
       "MyDistro" : { "64" : "ubuntu1304-amd64" }
     }
   },

   "Resources" : {

    "network1": {
      "Type": "OS::Quantum::Net"
    },

    "network2": {
      "Type": "OS::Quantum::Net"
    },

    "mgmt-subnet": {
      "Type": "OS::Quantum::Subnet",
      "Properties": {
        "network_id": { "Ref" : "network1" },
        "ip_version": 4,
        "cidr": "30.0.1.0/24",
"gateway_ip": "30.0.1.1",
"allocation_pools": [{"start": "30.0.1.2", "end": "30.0.1.20"}]
      }
    },

    "traffic-subnet": {
      "Type": "OS::Quantum::Subnet",
      "Properties": {
        "network_id": { "Ref" : "network2" },
        "ip_version": 4,
        "cidr": "40.0.1.0/24",
"gateway_ip": "40.0.1.1",
"allocation_pools": [{"start": "40.0.1.2", "end": "40.0.1.20"}]
      }
    },

    "network1port1": {
      "Type": "OS::Quantum::Port",
      "Properties": {
        "network_id": { "Ref" : "network1" }
      }
    },

    "network1port2": {
      "Type": "OS::Quantum::Port",
      "Properties": {
        "network_id": { "Ref" : "network1" }
      }
    },

    "network2port1": {
      "Type": "OS::Quantum::Port",
      "Properties": {
        "network_id": { "Ref" : "network2" }
      }
    },

    "network2port2": {
      "Type": "OS::Quantum::Port",
      "Properties": {
        "network_id": { "Ref" : "network2" }
      }
    },

     "inst1": {
       "Type": "AWS::EC2::Instance",
       "Metadata" : {
         "AWS::CloudFormation::Init" : {
         }
       },
       "Properties": {

         "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "MyDistribution" },
                           { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
         "InstanceType" : { "Ref" : "InstanceType" },
         "KeyName" : { "Ref" : "KeyName" },

   "NetworkInterfaces" : [
     { "NetworkInterfaceId" : { "Ref" : "network1port1" }, "DeviceIndex" : "0" },
     { "NetworkInterfaceId" : { "Ref" : "network2port1" }, "DeviceIndex" : "1" }
   ]

       }
     },

    "inst2": {
       "Type": "AWS::EC2::Instance",
       "Metadata" : {
         "AWS::CloudFormation::Init" : {
         }
       },
       "Properties": {

         "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" : "MyDistribution" },
                           { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
         "InstanceType" : { "Ref" : "InstanceType" },
         "KeyName" : { "Ref" : "KeyName" },
   "NetworkInterfaces" : [
     { "NetworkInterfaceId" : { "Ref" : "network1port2" }, "DeviceIndex" : "0" },
     { "NetworkInterfaceId" : { "Ref" : "network2port2" }, "DeviceIndex" : "1" }
   ]
       }
     }
  }
}

[Pekka Rinne (tsierkkis)]

It seems in this case after heat crash in nova there is no IP address at all listed for the instance. That is why heat crashes as it indexes using [0] without checking list length first. Why there is no IP i do not know, sometimes with same template IP is created ok - sometimes (most times) not. This then leads to empty list being sent to heat -> crash.

[Steve Baker (steve-stevebaker)]

I've confirmed this issue with the attached (slightly modified) template.

[Steve Baker (steve-stevebaker)]

The issue is that sometimes ports are being created before subnets, so there is no subnet to assign the ip from.

This was causing a different problem on stack delete, subnet deletes were failing because they still contained ports.

The fix is to create an implicit dependency from ports to subnets on the same network.

A code fix will be submitted soon, in the meantime you could confirm this by adding a DependsOn from each port to its subnet

eg
    "network1port1": {
      "Type": "OS::Quantum::Port",
     "DependsOn": "mgmt-subnet",
      "Properties": {
        "network_id": { "Ref" : "network1" }
      }
    },

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/34657

[Pekka Rinne (tsierkkis)]

I was testing with DependsOn set like proposed. It seems IP address is then allocated correctly.

But sometimes stack delete operation still fails because of "Failed to delete Subnet error. One or more ports have an IP allocation from this subnet". I just re-executed stack delete and then it went ok. Should DependsOn be in effect also in delete?

[Pekka Rinne (tsierkkis)]

Maybe this is related: I changed network2 to be external. After that maybe 1 out of 3 stack creations fails to: Bad router request: No IPs available for external network.

[Steve Baker (steve-stevebaker)]

Pekka could you raise 2 new issues for the above comments? And for #6 could you attach the template?

Thanks very much.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34657
Committed: http://github.com/openstack/heat/commit/f2ecb7fb6ec05e2040e36f9b015ae5684f06c14f
Submitter: Jenkins
Branch: master

commit f2ecb7fb6ec05e2040e36f9b015ae5684f06c14f
Author: Steve Baker <email address hidden>
Date: Thu Jun 27 13:39:30 2013 +1200

    Ports depend on the subnets in the same network.

    Without establishing this implicit dependency, the following
    sometimes occurs:
    - Ports get assigned to instances without any IP address
    - Stack delete fails as ports still exist in subnets.

    Fixes bug: #1192371

    Change-Id: Ifd892cef4c2eebaea5568afbd320c8865c845d70