Stack deletion failed because the VPCGatewayAttachment resource cannot be deleted

[Simon Pasquier]

A stack created with the following template cannot be deleted properly because Heat tries to delete the VPCGatewayAttachment resource before the EIP resource.

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description" : "Template for testing creation of VPC resources",

  "Parameters" : {

    "KeyName" : {
      "Description" : "Name of and existing EC2 KeyPair to enable SSH access to the instance",
      "Type" : "String"
    },

    "InstanceType" : {
      "Description" : "EC2 instance type",
      "Type" : "String",
      "Default" : "t1.micro",
      "AllowedValues" : [ "t1.micro", "m1.small", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium", "c1.xlarge", "cc1.4xlarge" ],
      "ConstraintDescription" : "must be a valid EC2 instance type."
    },

    "LinuxDistribution": {
      "Description" : "Distribution of choice",
      "Type": "String",
      "Default": "cirros-0.3.1-x86_64-uec"
    }

  },

  "Resources" : {

    "myVpc" : {
      "Type" : "AWS::EC2::VPC",
      "Properties" : {
        "CidrBlock" : "20.0.0.0/16"
      }
    },

    "mySubnet" : {
      "Type" : "AWS::EC2::Subnet",
      "Properties" : {
        "VpcId" : { "Ref" : "myVpc" },
        "CidrBlock" : "20.0.0.0/24"
      }
    },

    "myInternetGateway" : {
      "Type" : "AWS::EC2::InternetGateway"
    },

    "myInternetGatewayAssociation": {
      "Type" : "AWS::EC2::VPCGatewayAttachment",
      "Properties" : {
        "VpcId" : { "Ref" : "myVpc" },
        "InternetGatewayId" : { "Ref" : "myInternetGateway" }
      }
    },

    "IPAddress" : {
      "Type" : "AWS::EC2::EIP"
    },

    "IPAssoc" : {
      "Type" : "AWS::EC2::EIPAssociation",
      "Properties" : {
        "InstanceId" : { "Ref" : "myInstance" },
        "EIP" : { "Ref" : "IPAddress" }
      }
    },

    "InstanceSecurityGroup" : {
      "Type" : "AWS::EC2::SecurityGroup",
      "Properties" : {
        "GroupDescription" : "Enable ICMP + SSH access via port 22",
        "SecurityGroupIngress" : [
          {"IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "0.0.0.0/0"},
          {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0"}
        ]
      }
    },

    "myNetworkInterface": {
      "Type" : "AWS::EC2::NetworkInterface",
      "Properties" : {
        "SubnetId" : { "Ref" : "mySubnet" },
        "Description" : "myInstance interface"
      }
    },

    "myInstance" : {
      "Type" : "AWS::EC2::Instance",
      "DependsOn" : "InstanceSecurityGroup",
      "Properties" : {
        "ImageId" : { "Ref" : "LinuxDistribution" },
        "NetworkInterfaces" : [{ "NetworkInterfaceId": { "Ref" : "myNetworkInterface" }, "DeviceIndex" : "0" }],
        "KeyName" : { "Ref" : "KeyName" },
        "SecurityGroups" : [ {"Ref" : "InstanceSecurityGroup"} ],
        "InstanceType" : { "Ref": "InstanceType" }
      }
    }
  },

  "Outputs" : {
    "InstanceId" : {
      "Value" : { "Ref" : "myInstance" },
      "Description" : "Instance Id of newly created instance"
    },

    "IPAddress" : {
      "Value" : { "Ref" : "IPAddress" },
      "Description" : "Public IP address of instance"
    }
  }
}

$ heat stack-show myStack3
+----------------------+---------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+---------------------------------------------------------------------------------------------------------------------------------+
...
| stack_name | myStack3 |
| stack_status | DELETE_FAILED |
| stack_status_reason | Failed to delete VPCGatewayAttachment |
| | "myInternetGatewayAssociation" |
...
$ heat event-show myStack3 myInternetGatewayAssociation 188
...
| resource_status_reason | Gateway cannot be updated for router e3a99f2b-36c9-4c83-bbc8-0a2475f62e9b, since a gateway to external network 1b8a4b18-087f-4a1d-aac3-d0b64ac51e72 is required by one or more floating IPs. |
...

[Steve Baker (steve-stevebaker)]

Is there a stack trace from the heat-engine log that you can post for this?

[Simon Pasquier (simon-pasquier)]

Here is a log from heat-engine.
I cannot reproduce the error every time and sometimes I cannot start the stack because it fails at the creation of the EIPAssociation resource fails as the VPCGatewayAttachment resource is not created yet.
Looks like AWS::EC2::EIPAssociation and AWS::EC2::VPCGatewayAttachment resources have no dependency?

[Jeff Peeler (jpeeler-z)]

I think there are other resource deletion ordering problems that need to addressed when this is looked at. For example, I'm often able to reproduce subnet deletion errors due to the port not being deleted first:

2013-05-10 13:00:46.175 15500 ERROR heat.engine.resource [-] Delete Subnet "PrivateSubnet"
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource Traceback (most recent call last):
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 439, in delete
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource self.handle_delete()
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/subnet.py", line 95, in handle_delete
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource raise ex
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource QuantumClientException: Unable to complete operation on subnet 4af85e8c-947d-4cbd-a18f-727a9ee301f2. One or more ports have an IP allocation from this subnet.
2013-05-10 13:00:46.175 15500 TRACE heat.engine.resource
2013-05-10 13:00:46.203 15500 ERROR heat.engine.parser [-] Failed to delete Subnet "PrivateSubnet" error: QuantumClientException: Unable to complete operation on subnet 4af85e8c-947d-4cbd-a18f-727a9ee301f2. One or more ports have an IP allocation from this subnet.

[Simon Pasquier (simon-pasquier)]

It could be the same explanation than https://bugs.launchpad.net/heat/+bug/1176661/comments/5

[Jeff Peeler (jpeeler-z)]

This appears to have been fixed by http://github.com/openstack/heat/commit/4c1adb4c83f2f393dfb6a0460a8ecc3f14cd9b50

The code has changed a bit since this bug was filed. Now a router is created only when a RouteTable is associated with a given VPC. I haven't been able to reproduce any stack deletion failures yet.

[Steve Baker (steve-stevebaker)]

Cool, feel free to mark as Fixed Released - we can always reopen