Bug #1176661 “Stack creation fail when using Quantum floating IP...” : Bugs : OpenStack Heat

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-06:

#1

You have this:
    "floating_ip_DNSserver": {
      "Type": "OS::Quantum::FloatingIP",
      "Properties": {
        "floating_network_id": { "Ref" : "ExtNetUuid" }
      }
    },

It looks like ExtNetUuid is not reachable from the subnet you created, or you actually meant to do this in floating_ip_DNSserver:
"floating_network_id": { "Ref" : "network" }

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-06:

#2

No, I meant it the way it was written. ExtNetUuid is a parameter provided at launch time and is the UUID of admin created network that is used for external access. The template should create a local network, a router that is connected to this local network and to the external network. I also attached a screenshot of the network topology, but I am not sure if it got added.

If I try to recreate this stack manually, the situation is as follows. If I create local network and the three instances and then try to associate a floating IP, I will get the same error as above.If I first also create a router and connect it with the local network and the external network, associating a floating IP is successful. So it seems as if the stack creation tries to associate a floating IP before it creates the router from the template, which Quantum doesn't like and therefore it fails. If I watch the stack being created, the error happens before the router is created. If I remove the floating IP stuff, stack creatiion is successfull with local network, router and the three instances being created. After this I can also manually associate floating IPs.

Am i looking at this correctly or is there something that I misunderstand?

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-06:

#3

OK, I think I understand.

The order the resources are placed in the template has no determination on the order the resources are created.

Heat builds a dependency chain based on the dependencies it can see in the template (including Refs to other resources)

If there is an implicit dependency between resource not stated in the template you can always add an explicit DependsOn to ensure a resource is created first.

Something like this might help:
    "floating_ip_assoc_DNSserver": {
      "Type": "OS::Quantum::FloatingIPAssociation",
      "DependsOn" : "router",
      "Properties": {
        "floatingip_id": { "Ref" : "floating_ip_DNSserver" },
        "port_id": { "Ref" : "DNSServerPort" }
      }
    },

If this works we could investigate the association->router dependency being fixed in Heat

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-06: Re: [Bug 1176661] Re: Stack creation fail when using Quantum floating IPs

#4

Download full text (33.1 KiB)

I have added the following to all three associations:

      "DependsOn" : "router",
      "DependsOn" : "router_interface_private",
      "DependsOn" : "router_gateway_external",

I have added the second and third because I noticed that it created the
router, but it did not connect it
to both networks before creating an association and it failed again.

With the above three lines added it seems to be working most of the time,
but not always.

Event list working:

I have added the following to all three associations:

"DependsOn" : "router",
      "DependsOn" : "router_interface_private",
      "DependsOn" : "router_gateway_external",

I have added the second and third because I noticed that it created the
router, but it did not connect it
to both networks before creating an association and it failed again.

With the above three lines added it seems to be working most of the time,
but not always.

Event list working:

Not working:

saso@devstack-new-virt:~/diploma$ heat event-list DNS5
+-----------------------------+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+
| logical_resource_id         | id  | resource_status_reason

| resource_status |
event_time           |
+-----------------------------+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+
| floating_ip_Client2         | 529 | state changed

| CREATE_FAILED   |
2013-05-06T03:40:17Z |
+-----------------------------+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+
saso@devstack-new-virt:~/diploma$

Here in the not working case it seems it tried to create floating point
associations before it created router interface on the local network, even
though it has a line to depend on the
router_interface_private.

Stack delete also seems to fail sometimes, as in the case below:
saso@devstack-new-virt:~/diploma$ heat event-list DNS7
+-----------------------------+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+
| logical_resource_id         | id  | resource_status_reason

| resource_status    | event_time           |
+-----------------------------+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+
| floating_ip_Client1         | 746 | state changed

| DELETE_IN_PROGRESS | 2013-05-06T04:05:32Z |
| network                     | 821 | QuantumClientException: Unable to
complete operation on network 9bd22cb5-1fb0-4111-af7d-f75b98e4c14d. There
are one or more ports still in use on the network.
                   | DELETE_FAILED      | 2013-05-06T04:05:32Z |
+-----------------------------+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+----------------------+
saso@devstack-new-virt:~/diploma$

It seems stack deletion tries to delete router_interface_private before
releasing floating ips.

On Mon, May 6, 2013 at 4:39 AM, Steve Baker <sbaker@redhat.com> wrote:

> OK, I think I understand.
>
> The order the resources are placed in the template has no determination
> on the order the resources are created.
>
> Heat builds a dependency chain based on the dependencies it can see in
> the template (including Refs to other resources)
>
> If there is an implicit dependency between resource not stated in the
> template you can always add an explicit DependsOn to ensure a resource
> is created first.
>
> Something like this might help:
>     "floating_ip_assoc_DNSserver": {
>       "Type": "OS::Quantum::FloatingIPAssociation",
>       "DependsOn" : "router",
>       "Properties": {
>         "floatingip_id": { "Ref" : "floating_ip_DNSserver" },
>         "port_id": { "Ref" : "DNSServerPort" }
>       }
>     },
>
> If this works we could investigate the association->router dependency
> being fixed in Heat
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1176661
>
> Title:
>   Stack creation fail when using Quantum floating IPs
>
> Status in Orchestration API (Heat):
>   New
>
> Bug description:
>   I have created a template which uses Quantum resources and creates a
>   configuration with three instances on a private network, connected to
>   external network. It also assigns floating IPs to the three instances.
>   The problem is, that assigning floating IPs fails, because the router
>   is not yet created and connected before assigning floating IPs.
>
>   If I remove entries for floating IPs, the whole stack is successfully
> created.
>   The error reported by stack-show:
>   ...
>   | stack_status         | CREATE_FAILED
>
> |
>   | stack_status_reason  | Resource FloatingIPAssociation
>
>  |
>   |                      | "floating_ip_assoc_DNSserver" failed with:
>
>  |
>   |                      | QuantumClientException: External network
> 524ac1ab-f8d3
>        |
>   |                      | -47cd-b0df-cfdc7072b9cf is not reachable from
> subnet
>    |
>   |                      | 71145fd6-d5ce-4d10-8e91-46792a8b6e3a.
>  Therefore,
>           |
>   |                      | cannot associate Port 7c
>   ...
>
>   If I try to associate floating IPs from the dashboard and there is no
>   router connecting local and external nets, I get the same error.
>
>   Template:
>   {
>     "AWSTemplateFormatVersion" : "2013-05-04",
>
>     "Description" : "AWS CloudFormation template for use with OpenStack.
>   It uses Quantum for networking configuration. It sets up three Ubuntu
>   12.04 instances, each on its own subnet/network. One instance is set
>   up with bind9 for configuration of DNS server. When stack is created,
>   DNS is not configured, just installed on the instance. The other two
>   instances are used for testing DNS configuration. The three subnets
>   are connected with a router, that is also connected to the external
>   network. Floating IPs are also provided to the spawned instances for
>   external communication.",
>
>     "Parameters" : {
>       "KeyName" : {
>         "Description" : "Name of an existing EC2 KeyPair to enable SSH
> access to the instances",
>         "Type" : "String"
>       },
>
>       "ExtNetUuid" : {
>         "Description" : "UUID of the external network to be used for
> external access",
>         "Type" : "String"
>       },
>
>       "InstanceType" : {
>         "Description" : "DNSServer EC2 instance type",
>         "Type" : "String",
>         "Default" : "m1.micro",
>         "AllowedValues" : [ "t1.micro", "m1.small", "m1.large",
> "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium",
> "c1.xlarge", "cc1.4xlarge", "m1.micro" ],
>         "ConstraintDescription" : "must be a valid EC2 instance type."
>       },
>       "LinuxDistribution": {
>         "Default": "U12",
>         "Description" : "Distribution of choice",
>         "Type": "String",
>         "AllowedValues" : "U12"
>       }
>     },
>
>     "Mappings" : {
>       "AWSInstanceType2Arch" : {
>         "t1.micro"    : { "Arch" : "32" },
>         "m1.small"    : { "Arch" : "32" },
>         "m1.large"    : { "Arch" : "64" },
>         "m1.xlarge"   : { "Arch" : "64" },
>         "m2.xlarge"   : { "Arch" : "64" },
>         "m2.2xlarge"  : { "Arch" : "64" },
>         "m2.4xlarge"  : { "Arch" : "64" },
>         "c1.medium"   : { "Arch" : "32" },
>         "c1.xlarge"   : { "Arch" : "64" },
>         "cc1.4xlarge" : { "Arch" : "64" },
>         "m1.micro"    : { "Arch" : "64" }
>       },
>       "DistroArch2AMI": {
>         "U12"      : { "32" : "U12-i386-cfntools", "64" :
> "U12-x86_64-cfntools" }
>       }
>     },
>
>     "Resources" : {
>
>       "network": {
>         "Type": "OS::Quantum::Net",
>         "Properties": {
>           "name": "local_network"
>         }
>       },
>
>       "subnet": {
>         "Type": "OS::Quantum::Subnet",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "ip_version": 4,
>           "cidr": "10.0.10.0/24",
>           "allocation_pools": [{"start": "10.0.10.20", "end":
> "10.0.10.50"}]
>         }
>       },
>
>       "DNSServerPort": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.30"
>           }]
>         }
>       },
>
>       "Client1Port": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.31"
>           }]
>         }
>       },
>
>       "Client2Port": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.32"
>           }]
>         }
>       },
>
>       "router": {
>         "Type": "OS::Quantum::Router"
>       },
>
>       "router_interface_private": {
>         "Type": "OS::Quantum::RouterInterface",
>         "Properties": {
>           "router_id": { "Ref" : "router" },
>           "subnet_id": { "Ref" : "subnet" }
>         }
>       },
>
>       "router_gateway_external": {
>         "Type": "OS::Quantum::RouterGateway",
>         "Properties": {
>           "router_id": { "Ref" : "router" },
>           "network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "DNSServerSecurityGroup" : {
>         "Type" : "AWS::EC2::SecurityGroup",
>         "Properties" : {
>           "GroupDescription" : "Enable ping, SSH and DNS (port 53) access",
>           "SecurityGroupIngress" : [
>             {"IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "53", "ToPort" : "53",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "udp", "FromPort" : "53", "ToPort" : "53",
> "CidrIp" : "0.0.0.0/0"}
>           ]
>         }
>       },
>
>       "MinimalSecurityGroup" : {
>         "Type" : "AWS::EC2::SecurityGroup",
>         "Properties" : {
>           "GroupDescription" : "Enable only ping and SSH access",
>           "SecurityGroupIngress" : [
>             {"IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22",
> "CidrIp" : "0.0.0.0/0"}
>           ]
>         }
>       },
>
>       "DNSServer": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "services" : {
>                 "sysvinit" : {
>                   "bind9"    : { "enabled" : "true", "ensureRunning" :
> "true" }
>                 }
>               },
>               "packages" : {
>                 "apt" : {
>                   "bind9"    : [],
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "DNSServerSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "DNSServerPort" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "touch /tmp/cfninitwashere",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "Client1": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "packages" : {
>                 "apt" : {
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "MinimalSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "Client1Port" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "Client2": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "packages" : {
>                 "apt" : {
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "MinimalSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "Client2Port" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "floating_ip_DNSserver": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_Client1": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_Client2": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_assoc_DNSserver": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_DNSserver" },
>           "port_id": { "Ref" : "DNSServerPort" }
>         }
>       },
>
>       "floating_ip_assoc_Client1": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_Client1" },
>           "port_id": { "Ref" : "Client1Port" }
>         }
>       },
>
>       "floating_ip_assoc_Client2": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_Client2" },
>           "port_id": { "Ref" : "Client2Port" }
>         }
>       }
>     },
>
>     "Outputs" : {
>     }
>   }
>
>   Does the order of resources in the template matter, meaning would
>   changing the order have impact on resource creation?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/heat/+bug/1176661/+subscriptions
>

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-06:

#5

This sounds like a race where the router is not yet active when the next resources are created. The quantum resource types were written with the assumption that resources are active as soon as the create api call returns, but it looks like this assumption is not valid.

What quantum plugin are you using?

Each quantum resource type would benefit from having a check_active method written. I'll raise separate launchpad issues for each resource type.

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-06:

#6

Download full text (12.1 KiB)

I'm using standard Devstack install with Quantum. I think it uses
openVSwitch.

On Mon, May 6, 2013 at 6:55 AM, Steve Baker <email address hidden> wrote:

> This sounds like a race where the router is not yet active when the next
> resources are created. The quantum resource types were written with the
> assumption that resources are active as soon as the create api call
> returns, but it looks like this assumption is not valid.
>
> What quantum plugin are you using?
>
> Each quantum resource type would benefit from having a check_active
> method written. I'll raise separate launchpad issues for each resource
> type.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1176661
>
> Title:
> Stack creation fail when using Quantum floating IPs
>
> Status in Orchestration API (Heat):
> New
>
> Bug description:
> I have created a template which uses Quantum resources and creates a
> configuration with three instances on a private network, connected to
> external network. It also assigns floating IPs to the three instances.
> The problem is, that assigning floating IPs fails, because the router
> is not yet created and connected before assigning floating IPs.
>
> If I remove entries for floating IPs, the whole stack is successfully
> created.
> The error reported by stack-show:
> ...
> | stack_status | CREATE_FAILED
>
> |
> | stack_status_reason | Resource FloatingIPAssociation
>
> |
> | | "floating_ip_assoc_DNSserver" failed with:
>
> |
> | | QuantumClientException: External network
> 524ac1ab-f8d3
> |
> | | -47cd-b0df-cfdc7072b9cf is not reachable from
> subnet
> |
> | | 71145fd6-d5ce-4d10-8e91-46792a8b6e3a.
> Therefore,
> |
> | | cannot associate Port 7c
> ...
>
> If I try to associate floating IPs from the dashboard and there is no
> router connecting local and external nets, I get the same error.
>
> Template:
> {
> "AWSTemplateFormatVersion" : "2013-05-04",
>
> "Description" : "AWS CloudFormation template for use with OpenStack.
> It uses Quantum for networking configuration. It sets up three Ubuntu
> 12.04 instances, each on its own subnet/network. One instance is set
> up with bind9 for configuration of DNS server. When stack is created,
> DNS is not configured, just installed on the instance. The other two
> instances are used for testing DNS configuration. The three subnets
> are connected with a router, that is also connected to the external
> network. Floating IPs are also provided to the spawned instances for
> external communication.",
>
> "Parameters" : {
> "KeyName" : {
> "Description" : "Name of an existing EC2 KeyPair to enable SSH
> access to the instances",
> "Type" : "String"
> },
>
> "ExtNetUuid" : {
> "Description" : "UUID of the external network to be used for
> external access",
> "Type" : "String"
> },
>
> "InstanceType" : {
> "Description" : "DNSServer EC2 instance type",
> ...

I'm using standard Devstack install with Quantum. I think it uses
openVSwitch.

On Mon, May 6, 2013 at 6:55 AM, Steve Baker <sbaker@redhat.com> wrote:

> This sounds like a race where the router is not yet active when the next
> resources are created. The quantum resource types were written with the
> assumption that resources are active as soon as the create api call
> returns, but it looks like this assumption is not valid.
>
> What quantum plugin are you using?
>
> Each quantum resource type would benefit from having a check_active
> method written. I'll raise separate launchpad issues for each resource
> type.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1176661
>
> Title:
>   Stack creation fail when using Quantum floating IPs
>
> Status in Orchestration API (Heat):
>   New
>
> Bug description:
>   I have created a template which uses Quantum resources and creates a
>   configuration with three instances on a private network, connected to
>   external network. It also assigns floating IPs to the three instances.
>   The problem is, that assigning floating IPs fails, because the router
>   is not yet created and connected before assigning floating IPs.
>
>   If I remove entries for floating IPs, the whole stack is successfully
> created.
>   The error reported by stack-show:
>   ...
>   | stack_status         | CREATE_FAILED
>
> |
>   | stack_status_reason  | Resource FloatingIPAssociation
>
>  |
>   |                      | "floating_ip_assoc_DNSserver" failed with:
>
>  |
>   |                      | QuantumClientException: External network
> 524ac1ab-f8d3
>        |
>   |                      | -47cd-b0df-cfdc7072b9cf is not reachable from
> subnet
>    |
>   |                      | 71145fd6-d5ce-4d10-8e91-46792a8b6e3a.
>  Therefore,
>           |
>   |                      | cannot associate Port 7c
>   ...
>
>   If I try to associate floating IPs from the dashboard and there is no
>   router connecting local and external nets, I get the same error.
>
>   Template:
>   {
>     "AWSTemplateFormatVersion" : "2013-05-04",
>
>     "Description" : "AWS CloudFormation template for use with OpenStack.
>   It uses Quantum for networking configuration. It sets up three Ubuntu
>   12.04 instances, each on its own subnet/network. One instance is set
>   up with bind9 for configuration of DNS server. When stack is created,
>   DNS is not configured, just installed on the instance. The other two
>   instances are used for testing DNS configuration. The three subnets
>   are connected with a router, that is also connected to the external
>   network. Floating IPs are also provided to the spawned instances for
>   external communication.",
>
>     "Parameters" : {
>       "KeyName" : {
>         "Description" : "Name of an existing EC2 KeyPair to enable SSH
> access to the instances",
>         "Type" : "String"
>       },
>
>       "ExtNetUuid" : {
>         "Description" : "UUID of the external network to be used for
> external access",
>         "Type" : "String"
>       },
>
>       "InstanceType" : {
>         "Description" : "DNSServer EC2 instance type",
>         "Type" : "String",
>         "Default" : "m1.micro",
>         "AllowedValues" : [ "t1.micro", "m1.small", "m1.large",
> "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "c1.medium",
> "c1.xlarge", "cc1.4xlarge", "m1.micro" ],
>         "ConstraintDescription" : "must be a valid EC2 instance type."
>       },
>       "LinuxDistribution": {
>         "Default": "U12",
>         "Description" : "Distribution of choice",
>         "Type": "String",
>         "AllowedValues" : "U12"
>       }
>     },
>
>     "Mappings" : {
>       "AWSInstanceType2Arch" : {
>         "t1.micro"    : { "Arch" : "32" },
>         "m1.small"    : { "Arch" : "32" },
>         "m1.large"    : { "Arch" : "64" },
>         "m1.xlarge"   : { "Arch" : "64" },
>         "m2.xlarge"   : { "Arch" : "64" },
>         "m2.2xlarge"  : { "Arch" : "64" },
>         "m2.4xlarge"  : { "Arch" : "64" },
>         "c1.medium"   : { "Arch" : "32" },
>         "c1.xlarge"   : { "Arch" : "64" },
>         "cc1.4xlarge" : { "Arch" : "64" },
>         "m1.micro"    : { "Arch" : "64" }
>       },
>       "DistroArch2AMI": {
>         "U12"      : { "32" : "U12-i386-cfntools", "64" :
> "U12-x86_64-cfntools" }
>       }
>     },
>
>     "Resources" : {
>
>       "network": {
>         "Type": "OS::Quantum::Net",
>         "Properties": {
>           "name": "local_network"
>         }
>       },
>
>       "subnet": {
>         "Type": "OS::Quantum::Subnet",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "ip_version": 4,
>           "cidr": "10.0.10.0/24",
>           "allocation_pools": [{"start": "10.0.10.20", "end":
> "10.0.10.50"}]
>         }
>       },
>
>       "DNSServerPort": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.30"
>           }]
>         }
>       },
>
>       "Client1Port": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.31"
>           }]
>         }
>       },
>
>       "Client2Port": {
>         "Type": "OS::Quantum::Port",
>         "Properties": {
>           "network_id": { "Ref" : "network" },
>           "fixed_ips": [{
>             "subnet_id": { "Ref" : "subnet" },
>             "ip_address": "10.0.10.32"
>           }]
>         }
>       },
>
>       "router": {
>         "Type": "OS::Quantum::Router"
>       },
>
>       "router_interface_private": {
>         "Type": "OS::Quantum::RouterInterface",
>         "Properties": {
>           "router_id": { "Ref" : "router" },
>           "subnet_id": { "Ref" : "subnet" }
>         }
>       },
>
>       "router_gateway_external": {
>         "Type": "OS::Quantum::RouterGateway",
>         "Properties": {
>           "router_id": { "Ref" : "router" },
>           "network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "DNSServerSecurityGroup" : {
>         "Type" : "AWS::EC2::SecurityGroup",
>         "Properties" : {
>           "GroupDescription" : "Enable ping, SSH and DNS (port 53) access",
>           "SecurityGroupIngress" : [
>             {"IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "53", "ToPort" : "53",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "udp", "FromPort" : "53", "ToPort" : "53",
> "CidrIp" : "0.0.0.0/0"}
>           ]
>         }
>       },
>
>       "MinimalSecurityGroup" : {
>         "Type" : "AWS::EC2::SecurityGroup",
>         "Properties" : {
>           "GroupDescription" : "Enable only ping and SSH access",
>           "SecurityGroupIngress" : [
>             {"IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1",
> "CidrIp" : "0.0.0.0/0"},
>             {"IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22",
> "CidrIp" : "0.0.0.0/0"}
>           ]
>         }
>       },
>
>       "DNSServer": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "services" : {
>                 "sysvinit" : {
>                   "bind9"    : { "enabled" : "true", "ensureRunning" :
> "true" }
>                 }
>               },
>               "packages" : {
>                 "apt" : {
>                   "bind9"    : [],
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "DNSServerSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "DNSServerPort" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "touch /tmp/cfninitwashere",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "Client1": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "packages" : {
>                 "apt" : {
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "MinimalSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "Client1Port" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "Client2": {
>         "Type": "AWS::EC2::Instance",
>         "Metadata" : {
>           "AWS::CloudFormation::Init" : {
>             "config" : {
>               "packages" : {
>                 "apt" : {
>                   "dnsutils" : []
>                 }
>               }
>             }
>           }
>         },
>         "Properties": {
>           "ImageId" : { "Fn::FindInMap" : [ "DistroArch2AMI", { "Ref" :
> "LinuxDistribution" },
>                             { "Fn::FindInMap" : [ "AWSInstanceType2Arch",
> { "Ref" : "InstanceType" }, "Arch" ] } ] },
>           "InstanceType"   : { "Ref" : "InstanceType" },
>           "KeyName"        : { "Ref" : "KeyName" },
>           "SecurityGroups" : [ { "Ref" : "MinimalSecurityGroup" } ],
>           "NetworkInterfaces" : [ { "Ref" : "Client2Port" } ],
>           "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
>             "#!/bin/bash -v\n",
>             "/opt/aws/bin/cfn-init\n"
>           ]]}}
>         }
>       },
>
>       "floating_ip_DNSserver": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_Client1": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_Client2": {
>         "Type": "OS::Quantum::FloatingIP",
>         "Properties": {
>           "floating_network_id": { "Ref" : "ExtNetUuid" }
>         }
>       },
>
>       "floating_ip_assoc_DNSserver": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_DNSserver" },
>           "port_id": { "Ref" : "DNSServerPort" }
>         }
>       },
>
>       "floating_ip_assoc_Client1": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_Client1" },
>           "port_id": { "Ref" : "Client1Port" }
>         }
>       },
>
>       "floating_ip_assoc_Client2": {
>         "Type": "OS::Quantum::FloatingIPAssociation",
>         "Properties": {
>           "floatingip_id": { "Ref" : "floating_ip_Client2" },
>           "port_id": { "Ref" : "Client2Port" }
>         }
>       }
>     },
>
>     "Outputs" : {
>     }
>   }
>
>   Does the order of resources in the template matter, meaning would
>   changing the order have impact on resource creation?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/heat/+bug/1176661/+subscriptions
>

Steve Baker (steve-stevebaker) on 2013-05-07

Changed in heat:
status:	New → In Progress
importance:	Undecided → High
assignee:	nobody → Steve Baker (steve-stevebaker)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-05-07: Fix proposed to heat (master)

#7

Fix proposed to branch: master
Review: https://review.openstack.org/28377

Revision history for this message

Simon Pasquier (simon-pasquier) wrote on 2013-05-15:

#8

The same issue probably exists when deleting Quantum resources. See https://bugs.launchpad.net/heat/+bug/1165071

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-05-20: Fix merged to heat (master)

#9

Reviewed: https://review.openstack.org/28377
Committed: http://github.com/openstack/heat/commit/5d86e2f2ece51a79102aee2af9a3f306c929aaea
Submitter: Jenkins
Branch: master

commit 5d86e2f2ece51a79102aee2af9a3f306c929aaea
Author: Steve Baker <email address hidden>
Date: Tue May 7 16:00:55 2013 +1200

Implement check_active for quantum net, port, router

These quantum resources go through a BUILD status before becoming
ACTIVE (or DOWN).

This could explain the issues seen in
Bug: #1176661

Change-Id: I659774cd402b71be102376b31e2d78bf225d37a6

Changed in heat:
status:	In Progress → Fix Committed

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-20:

#10

Could you please update to the latest heat master, which has the above commit plus some other quantum fixes.

I would be very interested to know if this works now, and if you can remove the explicit DependsOn I asked you to add.

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-20:

#11

Download full text (34.4 KiB)

I have used a fresh devstack install with Quantum and OpenVswitch. I have checked git log for heat and the change seems to be present.

Unfortunately, the template with floating ips still keeps failing(I have removed DependsOn lines).
heat stack-show error line:
| stack_status | CREATE_FAILED |
| stack_status_reason | Resource FloatingIPAssociation |
| | "floating_ip_assoc_DNSserver" failed with: |
| | QuantumClientException: External network |
| | ea7c2afb-a520-4b96-b763-a638acd046bf is not reachable |
| | from subnet 727c8c8f-a836-4733-a690-0280bd94bce3. |
| | Therefore, cannot associate Port 59
stack event list:
saso@devstack-new-virt:~/diploma$ heat event-list DNS1
+-----------------------------+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+
| logical_resource_id | id | resource_status_reason | resource_status | event_time |
+-----------------------------+-----+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+
| router | 95 | state changed | IN_PROGRESS | 2013-05-20T05:12:10Z |
| router | 96 | state changed | CREATE_COMPLETE | 2013-05-20T05:12:10Z |
| network | 97 | state changed | IN_PROGRESS | 2013-05...

I have used a fresh devstack install with Quantum and OpenVswitch. I have checked git log for heat and the change seems to be present.

Out of 4 creation attempts one was successful, three were not. Even in the case where creation was successful, deletion was not.

Steve Baker (steve-stevebaker) on 2013-05-20

Changed in heat:
status:	Fix Committed → In Progress
milestone:	none → havana-1

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-20:

#13

I have reported the deleting issue as a separate bug here:

https://bugs.launchpad.net/heat/+bug/1182266

The ext_net I pass as a parameter looks like this:

saso@devstack-new-virt:~/diploma$ quantum subnet-show ext_subnet
+------------------+--------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.1.60", "end": "192.168.1.70"} |
| cidr | 192.168.1.0/24 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.1.100 |
| host_routes | |
| id | 02538e8d-530a-4117-939e-10491c55b0fa |
| ip_version | 4 |
| name | ext_subnet |
| network_id | 3c130c14-6dad-4fc8-9927-0fe2de4e3c1d |
| tenant_id | 38fcc864314946e68c1030a2180c19e2 |
+------------------+--------------------------------------------------+
saso@devstack-new-virt:~/diploma$

I have reported the deleting issue as a separate bug here:

https://bugs.launchpad.net/heat/+bug/1182266

The ext_net I pass as a parameter looks like this:

saso@devstack-new-virt:~/diploma$ quantum subnet-show ext_subnet
+------------------+--------------------------------------------------+
| Field            | Value                                            |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.1.60", "end": "192.168.1.70"} |
| cidr             | 192.168.1.0/24                                   |
| dns_nameservers  |                                                  |
| enable_dhcp      | False                                            |
| gateway_ip       | 192.168.1.100                                    |
| host_routes      |                                                  |
| id               | 02538e8d-530a-4117-939e-10491c55b0fa             |
| ip_version       | 4                                                |
| name             | ext_subnet                                       |
| network_id       | 3c130c14-6dad-4fc8-9927-0fe2de4e3c1d             |
| tenant_id        | 38fcc864314946e68c1030a2180c19e2                 |
+------------------+--------------------------------------------------+
saso@devstack-new-virt:~/diploma$

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-20:

#14

I have replicated both the create and delete issue locally using the template in the description of this bug. I have a fix which appears to work for create and delete.

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-21:

#15

Will the fix be committed into git?

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-05-21:

#17

Please take a look at this review
https://review.openstack.org/#/c/29857/

You may wish to join the review discussion on whether OS::Quantum::FloatingIP should specify a gateway property

Until the fix is committed, you can achieve the same effect with the following DependsOn.

    "floating_ip_DNSserver": {
      "Type": "OS::Quantum::FloatingIP",
      "DependsOn": "router_gateway_external",
      "Properties": {
        "floating_network_id": { "Ref" : "ExtNetUuid" }
      }
    },

    "floating_ip_Client1": {
      "Type": "OS::Quantum::FloatingIP",
      "DependsOn": "router_gateway_external",
      "Properties": {
        "floating_network_id": { "Ref" : "ExtNetUuid" }
      }
    },

    "floating_ip_Client2": {
      "Type": "OS::Quantum::FloatingIP",
      "DependsOn": "router_gateway_external",
      "Properties": {
        "floating_network_id": { "Ref" : "ExtNetUuid" }
      }
    },

Revision history for this message

Saso Kavcic (sasokavcic66) wrote on 2013-05-22:

#18

I have tested stack creation with the above workaround and it solves my problem. I have made five attempts and each and every one of them was successful.

Steven Hardy (shardy) on 2013-05-28

Changed in heat:
milestone:	havana-1 → havana-2

Steve Baker (steve-stevebaker) on 2013-06-27

Changed in heat:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2013-07-17

Changed in heat:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2013-10-17

Changed in heat:
milestone:	havana-2 → 2013.2

OpenStack Heat

Stack creation fail when using Quantum floating IPs

Bug Description

Other bug subscribers

Remote bug watches