stack delete failed when using OS::Neutron::Router's external_gateway_info property

Bug #1279645 reported by Mitsuru Kanabuchi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Dong Liu

Bug Description

In case of using OS::Neutron::Router's external_gateway_info property, stack delete failed with Neutron 409 error.

procedure for reproducing

  1) stack create with template file [1]
  2) check CREATE_COMPLETE
  3) execute stack delete
  4) stack delete failed with Neutron 409 'SubnetInUse' error, heat-engine's log pasted [2]

[1] template file

  {
    "AWSTemplateFormatVersion" : "2010-09-09",
    "Resources" : {
      "net_external": {
        "Type": "OS::Neutron::Net",
        "Properties": {
          "name": "net_external",
          "admin_state_up": true,
          "value_specs": {
            "provider:network_type": "flat",
            "provider:physical_network": "default",
            "router:external": true
          }
        }
      },
      "subnet_external": {
        "Type": "OS::Neutron::Subnet",
        "Properties": {
          "name": "subnet_external",
          "network_id": { "Ref" : "net_external" },
          "ip_version": 4,
          "cidr": "192.168.10.0/24",
          "gateway_ip": "192.168.10.11",
          "enable_dhcp": false,
          "allocation_pools": [ { "start": "192.168.10.200", "end": "192.168.10.253" } ]
        }
      },
      "router": {
        "Type": "OS::Neutron::Router",
        "Properties": {
          "name": "router",
          "external_gateway_info": {
            "network": { "Ref" : "net_external" }
          }
        }
      }
    }
  }

[2]heat-engine's error log

  2014-02-13 13:04:09.499 ERROR heat.engine.resource [-] Delete Subnet "subnet_external" [bf4b80c4-de29-4416-8bd9-3258447f02e5] Stack "external_gateway_info" [20e71a8a-7207-49b8-ba99-a5d4c5dd096d]
  2014-02-13 13:04:09.499 TRACE heat.engine.resource Traceback (most recent call last):
  2014-02-13 13:04:09.499 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 695, in delete
  2014-02-13 13:04:09.499 TRACE heat.engine.resource handle_data = self.handle_delete()
  2014-02-13 13:04:09.499 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/neutron/subnet.py", line 150, in handle_delete
  2014-02-13 13:04:09.499 TRACE heat.engine.resource self._handle_not_found_exception(ex)
  2014-02-13 13:04:09.499 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/neutron/neutron.py", line 141, in _handle_not_found_exception
  2014-02-13 13:04:09.499 TRACE heat.engine.resource raise ex
  2014-02-13 13:04:09.499 TRACE heat.engine.resource NeutronClientException: 409-{u'NeutronError': {u'message': u'Unable to complete operation on subnet bf4b80c4-de29-4416-8bd9-3258447f02e5. One or more ports have an IP allocation from this subnet.', u'type': u'SubnetInUse', u'detail': u''}}
  2014-02-13 13:04:09.499 TRACE heat.engine.resource

Dong Liu (liudong78)
Changed in heat:
assignee: nobody → Dong Liu (liudong78)
Revision history for this message
Dong Liu (liudong78) wrote :

I create stack with this template, it was failed sometimes for "No IPs available for external network".
I think this is because heat create router before create subnet.
So maybe we should add "DependsOn": "subnet_external" property in router resource.
I test this, and it never failed.

Revision history for this message
Mitsuru Kanabuchi (kanabuchi) wrote :

I guess you're right. However, a lot of users won't notice that DependsOn is necessary when using external_gateway_info.

On the one hand, it's no problem without DependsOn when using OS::Neutron::RouterGateway.
I think, adding following codes to OS::Neutron::Router would avoid this problem.

https://github.com/openstack/heat/blob/master/heat/engine/resources/neutron/router.py#L273

I'm not sure what it really should add. But it's better for user experience.

Dong Liu (liudong78)
Changed in heat:
assignee: Dong Liu (liudong78) → nobody
Revision history for this message
Dong Liu (liudong78) wrote :

RouterGateway is deprecated, I considered add dependence in Router resource, but it seems not a good way.

I have no better thought about this.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I thought the intent of adding external_gateway_info was to avoid the races that RouterGateway had a workaround for.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I wonder if this is needed in Router handle_delete before delete_router is called:

  if props.get(self.EXTERNAL_GATEWAY):
        try:
            client.remove_gateway_router(self.resource_id)
        except NeutronClientException as ex:
            self._handle_not_found_exception(ex)

Changed in heat:
milestone: none → icehouse-rc1
status: New → Triaged
importance: Undecided → Medium
Changed in heat:
assignee: nobody → Bartosz Górski (bartosz-gorski)
Revision history for this message
Dong Liu (liudong78) wrote :

Neutron will remove router gatway when deleting router, so I think it is no needed to add client.remove_gateway_router(self.resource_id) before delete_router.

In this issue, I think heat deleted subnet before deleted router, so this router-gateway port was in used, subnet deletion was failed.

And there is still another issue that the stack creation maybe failed because create router with gateway need at least one subnet in external network.

I created this stack some times, it was failed probabilistic as heat create router before create subnet.

I think add dependence is necessary.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Another option is modifying Subnet handle_delete to add a task which polls for associated ports until that reaches zero. I might attempt this approach to fix both this and bug 1243992

Revision history for this message
Zane Bitter (zaneb) wrote :

The intent of adding external_gateway_info was to avoid the races where other resources had to have a workaround to depend on RouterGateway. (afaict this is just the FloatingIP resource.)

So if we copied the hack from https://github.com/openstack/heat/blob/master/heat/engine/resources/neutron/router.py#L273 then we are no worse off than before, though it would be nice to be better.

What would really be nice is if the router-gateway property referenced a Subnet instead of a network. But this is not how the underlying API works - that seems to just silently pick a subnet (which one? how?) from the network provided and silently add an interface on it - an interface you can't find or access if you wanted to refer to it when creating, say an ExtraRoute, but which nevertheless stops you from deleting the subnet.

The more I see of the Neutron API, the less I understand it :(

Revision history for this message
Bartosz Górski (bartosz-gorski) wrote :

I unassinged myself because Dong have a fix proposition and wants to take it.

Changed in heat:
assignee: Bartosz Górski (bartosz-gorski) → nobody
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Maybe there is an assumption that any network which could be used for an external gateway will have only one subnet.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/81976

Changed in heat:
assignee: nobody → Steve Baker (steve-stevebaker)
status: Triaged → In Progress
Revision history for this message
Steve Baker (steve-stevebaker) wrote :

My check for ports on delete approach won't work, there are races on create too where the subnet is expected to exist.

Changed in heat:
assignee: Steve Baker (steve-stevebaker) → Dong Liu (liudong78)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/82452

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/82452
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=1ed544eb80e8a92a67188c53897342d843a20e7a
Submitter: Jenkins
Branch: master

commit 1ed544eb80e8a92a67188c53897342d843a20e7a
Author: liudong <email address hidden>
Date: Mon Mar 24 15:17:39 2014 +0800

    Add subnets as a dependency for router

    Because router's external gateway depends on at least one external subnet,
    so add subnets as an implicit dependency for the router.

    Change-Id: Ib34636b440b3eee3993cd1030dab23312824510d
    Closes-Bug:1279645

Changed in heat:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.