Fail to add dependency of port and subnet when fixed_ip is not set

[Haiwei Xu]

Previously when creating a floating_ip without setting the fixed_ip, the stack can not be deleted, the bug report is here:
https://bugs.launchpad.net/heat/+bug/1428434
This bug is fixed, but not perfectly. Currently when creating a floating_ip without setting the fixed_ip, will get this error:

2015-04-09 20:29:50.895 DEBUG neutronclient.client [req-001dd880-d78b-4ecc-98ec-4308b1c8ccac None 5e35ad29e19846469182ea649554a80f-84bc7968-b554-4d5c-ba60-5d
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/common/context.py", line 304, in wrapped
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/service.py", line 1104, in describe_stack_resource
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher with_attr=with_attr)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/api.py", line 176, in format_stack_resource
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher rpc_api.RES_REQUIRED_BY: resource.required_by(),
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/resource.py", line 436, in required_by
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher [r.name for r in self.stack.dependencies.required_by(self)])
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/stack.py", line 241, in dependencies
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher self.resources.itervalues())
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/stack.py", line 328, in _get_dependencies
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher res.add_dependencies(deps)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/resources/openstack/neutron/floatingip.py", line 170, in add_dependencies
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher if port_on_subnet(d, interface_subnet):
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/heat/heat/engine/resources/openstack/neutron/floatingip.py", line 150, in port_on_subnet
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher self.neutron().show_network(p_net)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher ret = self.function(instance, *args, **kwargs)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 531, in show_network
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher return self.get(self.network_path % (network), params=_params)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 292, in get
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher headers=headers, params=params)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 268, in retry_request
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher headers=headers, params=params)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 209, in do_request
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher self._handle_fault_response(status_code, replybody)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher exception_handler_v20(status_code, des_error_body)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 67, in exception_handler_v20
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher status_code=status_code)
2015-04-09 20:29:50.896 TRACE oslo_messaging.rpc.dispatcher NetworkNotFoundClient: Network ab6a1101-5bd0-4fb2-9de3-480d0f46af21 could not be found

"ab6a1101-5bd0-4fb2-9de3-480d0f46af21" network is created but can not be seen by user.

Here is part of the template:

  server:
    type: OS::Nova::Server
    properties:
      image: {get_param: image}
      flavor: {get_param: flavor}
      key_name: {get_param: key_name}
      name: puppet_inst
      user_data_format: SOFTWARE_CONFIG
      networks:
        - port: { get_resource: multisoftpuppet_port }
  multisoftpuppet_port:
    type: OS::Neutron::Port
    properties:
      network: { get_resource: multisoft_network }
      security_groups: [ get_resource: the_sg ]

  multisoft_floating_ip:
    type: OS::Neutron::FloatingIP
    properties:
      floating_network: { get_param: public_network }
      port_id: { get_resource: multisoftpuppet_port }

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/173237

[Sergey Kraynev (skraynev)]

I tried to create stack with attached template and it work for me correct, so I can not reproduce your issue.
Could you please add more details how to reproduce it.

[Haiwei Xu (xu-haiwei)]

Yes, your template works for me also, but mine does not work. Maybe I am using SoftwareConfig?

[Steve Baker (steve-stevebaker)]

Haiuwei, if you are creating the subnet resource in your template then your port should reference the subnet resource instead of (or as well as) the network resource.

This is because ports are actually associated with subnets, so heat can properly model the neutron dependencies.

Basically you should use subnet references (or depends_on) wherever you need to to ensure the subnet is deleted after all the ports created on it.

[Haiwei Xu (xu-haiwei)]

Steve Baker

Thank you for reply. Like you have said, the template is wrong. Configure a fixed_ip to either Port resource or FloatingIP resource will avoid this error.

But if user missed the fixed_ip the error will still happen, besides leading the users to write the correct template, is there any other way to avoid the error?

[OpenStack Infra (hudson-openstack)]

Change abandoned by xu-haiwei (<email address hidden>) on branch: master
Review: https://review.openstack.org/173237

[Johannes Grassler (jgr-launchpad)]

Regardless of the proper way to associate ports with subnets, people will get it wrong, leaving broken stacks in their wake when stack-delete fails due this kind of race condition (current example in our cloud: a stack gets stuck in state DELETE_IN_PROGRESS, `heat resource-list` on the stack blows up with a NetworkNotFoundClient exception and `heat stack-abandon` fails as well).

As far as I can see, adding exception handling in places like https://github.com/openstack/heat/blob/master/heat/engine/resources/openstack/neutron/floatingip.py#L169 should go a long way towards ameliorating the problem: if a Neutron resource (such as a network) no longer exists, it needn't be added as a dependency either. Is my reasoning correct there, or are there some side effects I might be overlooking?

The current state of affairs leaves something to be desired, at any rate: one cannot even attempt to clean up manually if somebody instantiates a broken template, because this lack of exception handling breaks resource-list, effectively blinding the operator (short of plucking resource IDs from Heat's database, that is).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/289371

[Johannes Grassler (jgr-launchpad)]

I had a look at the other Neutron resources now and there is only one other potential trouble spot where the same kind of thing (looking up a resource ID from Neutron without exception handling) happens:

https://github.com/openstack/heat/commit/547457828ded4c4d8ecec1546703cd154b8e4a8c#diff-f62d2ab8ef023c0a756f83e1209d1e9cR62

At first glance it does not appear to be related to that commit's message, but I might be wrong. Is that lookup really neccessary? As far as I understand (correct me if I'm wrong) it yields the same Router ID that is already recorded in the RouterInterface's properties. If it is required I'll add exception handling in that spot as well: after all a RouterInterface with no associated Router is functionally equivalent to a RouterInterface with no associated Subnet. It does not need to be part of the dependency graph since it is just a piece of stale information in Heat's database.

[Zane Bitter (zaneb)]

Per discussion on https://review.openstack.org/#/c/289371/8 I think we're going to aim to just catch any exceptions that occur for this bug, and I've opened several other bugs (bug 1626607, bug 1626619, bug 1626630, bug 1626634) to address the wider issue of getting rid of ReST calls from the dependency calculations altogether.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/489276

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zane Bitter (<email address hidden>) on branch: master
Review: https://review.openstack.org/289371
Reason: OK, time to abandon this I think.

Several of the other bugs I opened have already been fixed (bug 1626619, bug 1626607).

I just submitted a patch to resolve the original bug (bug 1442121), constrained to the scope we decided on in the comments here: Ic9220041e7bf85b4388ad283082833cfbdb6c76a.

The others (bug 1626630 and bug 1626634) remain open, but the eventual solutions will likely not look much like the ones proposed in this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rabi Mishra <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/heat/+/424033