FloatingIP needs a hidden dependency on RouterInterfaces

[Zane Bitter]

I'm rewriting the bug description to match the bug that was actually fixed, which is discussed in comment #1.

The OS::Neutron::FloatingIP resource requires an implicit dependency on any OS::Neutron::RouterInterface resources where the interface connects INTERNAL subnet to which the FloatingIP is connected to the router that provides the external gateway.

Original text follows. The original issue has been moved to bug 1399699.

---

The OS::Neutron::FloatingIP resource has an implicit dependency on any OS::Neutron::RouterGateway resources where the gateway target is the same Network on which the FloatingIP is allocated, since a Router on the Subnet containing the Port must be connected to the Gateway before a FloatingIP can be allocated to it.

However, OS::Neutron::RouterGateway has been deprecated in favour of a property on the OS::Neutron::Router resource. So we also need the same implicit dependency on Routers with the FloatingIP's network specified as the external gateway target.

[Zane Bitter (zaneb)]

After discussion on IRC of another bug, it appears that depending on the Router (and by extension the gateway) is insufficient, and we need to depend instead on the RouterInterface that connects the Router to the internal Subnet.

This suggests that it's probably a mistake to have the RouterInterfaces take a Router argument, rather than the other way around; if Router took a list of Ports instead then depending on the Router would be sufficient.

[Steven Hardy (shardy)]

No fix proposed, so bumping to J3

[Jacques Uber (uberj)]

I'm attaching a template that I can reproduce this bug with. You must have two instances consuming port and floating ip resources for the bug to manifest itself. For some reason removing one of the servers will cause the deletion to happen without issue. I haven't tried it with three.

The command I'm invoking the template with is:

    heat stack-create dpaste --template-file=bug_template.yaml --parameters="key_name=uberj-key;public_net_id=7b3170a7-3c5f-465f-8bbf-6dd53a3ac6ae" -D

7b3170a7-3c5f-465f-8bbf-6dd53a3ac6ae is a reference to a shared external neutron network.

[Steve McLellan (sjmc7)]

I've experienced the same issue with a similar template to uberj, although we can reproduce it with one server, and we don't create the router (it's provided as an external id).

Suggested by Steve Baker, adding a 'depends_on' from the floating ip to the router interface fixes the issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/122592

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/122592
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5c158edac6950899c41d3090cde8c636e63c4a2f
Submitter: Jenkins
Branch: master

commit 5c158edac6950899c41d3090cde8c636e63c4a2f
Author: Steve Baker <email address hidden>
Date: Fri Sep 19 12:18:12 2014 +1200

    Associate floating IP with router interface

    This change will create a dependency from a FloatingIP to a RouterInterface in
    this template which interfaces with the same subnet that this FloatingIP's
    port is assigned to.

    It would be preferable to add the dependency based on matching FloatingIP
    floating_network_id and Router external_gateway_info network, but there is
    a valid use-case for the Router being external to the template, so the
    dependency is matched on the internal subnet instead, which is available
    from the RouterInterface property.

    Change-Id: Iedff00b382b4fcda741ca5c9b4adc23b176ec48c
    Closes-Bug: #1299259

[Igor Milovanović (igormilovanovic)]

I'm still seeing this running Juno, when I try to delete a stack via Horizon:

2014-12-02 18:21:35.800 5084 INFO heat.engine.resource [-] deleting RouterInterface "router1_interface" [4a392766-5eb6-4854-b896-85d355e82ee6:subnet_id=ddf1b745-7eba-4063-a5a9-491283da7750] Stack "buss_stack" [2f27941a-95d1-42d5-ae96-30063beaa4c2]
2014-12-02 18:21:35.825 5084 DEBUG neutronclient.client [-]
REQ: curl -i http://172.19.29.167:9696//v2.0/routers/4a392766-5eb6-4854-b896-85d355e82ee6/remove_router_interface.json -X PUT -H "X-Auth-Token: ac1e5a57a784e566825828906b2be285" -H "User-Agent: python-neutronclient" -d '{"subnet_id": "ddf1b745-7eba-4063-a5a9-491283da7750"}'
 http_log_req /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:140
2014-12-02 18:21:35.826 5084 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 172.19.29.167
2014-12-02 18:21:35.954 5084 DEBUG urllib3.connectionpool [-] "PUT //v2.0/routers/4a392766-5eb6-4854-b896-85d355e82ee6/remove_router_interface.json HTTP/1.1" 409 268 _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:357
2014-12-02 18:21:35.954 5084 DEBUG neutronclient.client [-] RESP:409 {'date': 'Tue, 02 Dec 2014 18:21:35 GMT', 'content-length': '268', 'content-type': 'application/json; charset=UTF-8', 'x-openstack-request-id': 'req-4102ee51-d655-497b-bd78-aad277deb86a'} {"NeutronError": {"message": "Router interface for subnet ddf1b745-7eba-4063-a5a9-491283da7750 on router 4a392766-5eb6-4854-b896-85d355e82ee6 cannot be deleted, as it is required by one or more floating IPs.", "type": "RouterInterfaceInUseByFloatingIP", "detail": ""}}
 http_log_resp /usr/lib/python2.7/site-packages/neutronclient/common/utils.py:149
2014-12-02 18:21:35.955 5084 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Router interface for subnet ddf1b745-7eba-4063-a5a9-491283da7750 on router 4a392766-5eb6-4854-b896-85d355e82ee6 cannot be deleted, as it is required by one or more floating IPs.", "type": "RouterInterfaceInUseByFloatingIP", "detail": ""}} _handle_fault_response /usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py:1207
2014-12-02 18:21:35.955 5084 INFO heat.engine.resource [-] DELETE: RouterInterface "router1_interface" [4a392766-5eb6-4854-b896-85d355e82ee6:subnet_id=ddf1b745-7eba-4063-a5a9-491283da7750] Stack "buss_stack" [2f27941a-95d1-42d5-ae96-30063beaa4c2]
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource Traceback (most recent call last):
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 435, in _action_recorder
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource yield
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 839, in delete
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource yield self.action_handler_task(action, *action_args)
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 286, in wrapper
2014-12-02 18:21:35.955 5084 TRACE heat.engine.resource step = next(subtask)
2014-12-02 18:21:35...

[Igor Milovanović (igormilovanovic)]

Template used to create this stack

[Zane Bitter (zaneb)]

Igor,
The reason it's failing is that the port is obtained from {get_attr: [lb_buss_1_lb_pool, vip, port_id]} rather than from a OS::Neutron::Port resource created in the same template. There's no (efficient) way for Heat to determine whether this port is on the same subnet as the router interface, so this problem is unfixable with the current data model. The solution is to add an explicit dependency yourself:

  lb_001-lb-fip:
    properties:
      floating_network: public
      port_id:
        get_attr: [lb_buss_1_lb_pool, vip, port_id]
    depends_on:
      - router1_interface
    type: OS::Neutron::FloatingIP

[Igor Milovanović (igormilovanovic)]

Hi Zane,

thank you for the explanation.

Cheers!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139607

[Zane Bitter (zaneb)]

Wow, this turned out to be super confusing :/

So there are actually 3 bugs:

1) The originally reported issue, a dependency on the EXTERNAL network. I have now raised bug 1399699 for that, since it still isn't fixed.
2) The issue we decided to fix in the discussion documented in comment #1, a dependency on the INTERNAL network. I'm leaving that one here because it did actually get fixed in Juno.
3) The same issue that was fixed here for FloatingIPs, except on FloatingIPAssociations. I have now raised bug 1399702 for that one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Kraynev (<email address hidden>) on branch: master
Review: https://review.openstack.org/139607
Reason: I have checked the same situation for another order resources for add_dependencies. So it looks correct and I can not reproduce error now.
So it's abondoned in favor of https://review.openstack.org/#/q/status:open+project:openstack/heat+branch:master+topic:add_dependencies,n,z