OS::Neutron::FloatingIP is missing hidden dependencies

Bug #1626619 reported by Zane Bitter
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Zane Bitter
Newton
Fix Committed
Undecided
Rabi Mishra

Bug Description

FloatingIPs depend on RouterInterfaces that are on the same subnet that the Port holding the FloatingIP is attached to. (IOW you can't create a FloatingIP until it has some path to get to the outside world via a Router.) Originally this was checked simply by finding the Port resource, then looking through its `fixed_ips` property looking for `subnet` or subnet_id` subproperties that matched the `subnet` or `subnet_id` properties of the RouterInterface.

This changed in https://review.openstack.org/#/c/167782/3/heat/engine/resources/openstack/neutron/floatingip.py to additionally handle the case where the `fixed_ips` property of the Port is not set. In this case we have no way of knowing which subnet the port will land on (depending on whom you believe, it is either assigned to a subnet chosen behind the scenes by Neutron or assigned to _all_ subnets). Therefore in this case we add all RouterInterfaces on any of the Subnets in the same Network.

To do so we make an expensive API call (for every RouterInterface!) to get the list of subnets in the network. This is painful but not crazy, since it's solving a real issue (bug 1428434).

The same patch also attempts to fix the issue with adding too many dependencies during create, unfortunately by causing the opposite problem: not adding _any_ dependencies during create. That kind-of worked prior to convergence, when the dependencies were recalculated for each new stack operation. That mean that stuff would be deleted in the right order. However, under convergence the dependencies are stored in the database as soon as the stack is created, so we will see a recurrence of deletes failing because of missing implicit dependencies.

The solution here is to change it back to depending on all RouterInterfaces at create time, instead of none of them.

See further discussion in https://review.openstack.org/#/c/289371/ for context.

Zane Bitter (zaneb)
description: updated
Revision history for this message
Zane Bitter (zaneb) wrote :

I raised bug 1626634 for the issue of paring this back again to the minimum number of dependencies that we need, and getting rid of the RPC call.

Revision history for this message
Zane Bitter (zaneb) wrote :

Er, s/RPC/ReST/ in that last comment.

Changed in heat:
assignee: nobody → Oleksii Chuprykov (ochuprykov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/381219

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/393922

Changed in heat:
assignee: Oleksii Chuprykov (ochuprykov) → Zane Bitter (zaneb)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/393922
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=d5ef455f660d342f4eb8e2a5a9531d4fc30ac25c
Submitter: Jenkins
Branch: master

commit d5ef455f660d342f4eb8e2a5a9531d4fc30ac25c
Author: Zane Bitter <email address hidden>
Date: Fri Nov 4 16:06:43 2016 -0400

    Make FloatingIP depend on all RouterInterfaces again

    It used to be that a FloatingIP would depend on all RouterInterfaces that
    it couldn't determine were on a different subnet to the port the FloatingIP
    was attaching to (by virtue of the Subnet value being None in both cases,
    and therefore matching). That changed in
    c9f32bfef08e567fda6848686ff6281fa3344c69 to depending only on
    RouterInterfaces that it *could* determine *were* in the same subnet. This
    resulted in no dependency being added at create time when the Subnet was a
    resource in the local stack (as opposed to being passed in e.g. as a
    parameter).

    That caused problems with some Neutron plugins, but worked with others
    where the dependencies were required only at delete time (at this point the
    in-template Subnets had IDs and therefore the dependencies were fully
    determined). However, with convergence phase 1 enabled, the dependencies
    calculated at create time are used throughout the lifecycle of the stack,
    thus causing non-deterministic ordering problems at delete time.

    This patch restores the behaviour where if the Subnet for a RouterInterface
    cannot be determined, the FloatingIP will depend on it. It also corrects a
    bug where we only checked the first Subnet if multiple Networks/Subnets
    were listed for the port.

    Change-Id: I185e30b2247c5b423d319506f60864c3027fc4fe
    Closes-Bug: #1626619

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/394814

Rabi Mishra (rabi)
tags: removed: newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/newton)

Reviewed: https://review.openstack.org/394814
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ae3b079a46811663d20998e8b5d31ad79f3242db
Submitter: Jenkins
Branch: stable/newton

commit ae3b079a46811663d20998e8b5d31ad79f3242db
Author: Zane Bitter <email address hidden>
Date: Fri Nov 4 16:06:43 2016 -0400

    Make FloatingIP depend on all RouterInterfaces again

    It used to be that a FloatingIP would depend on all RouterInterfaces that
    it couldn't determine were on a different subnet to the port the FloatingIP
    was attaching to (by virtue of the Subnet value being None in both cases,
    and therefore matching). That changed in
    c9f32bfef08e567fda6848686ff6281fa3344c69 to depending only on
    RouterInterfaces that it *could* determine *were* in the same subnet. This
    resulted in no dependency being added at create time when the Subnet was a
    resource in the local stack (as opposed to being passed in e.g. as a
    parameter).

    That caused problems with some Neutron plugins, but worked with others
    where the dependencies were required only at delete time (at this point the
    in-template Subnets had IDs and therefore the dependencies were fully
    determined). However, with convergence phase 1 enabled, the dependencies
    calculated at create time are used throughout the lifecycle of the stack,
    thus causing non-deterministic ordering problems at delete time.

    This patch restores the behaviour where if the Subnet for a RouterInterface
    cannot be determined, the FloatingIP will depend on it. It also corrects a
    bug where we only checked the first Subnet if multiple Networks/Subnets
    were listed for the port.

    Change-Id: I185e30b2247c5b423d319506f60864c3027fc4fe
    Closes-Bug: #1626619
    (cherry picked from commit d5ef455f660d342f4eb8e2a5a9531d4fc30ac25c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 8.0.0.0b1

This issue was fixed in the openstack/heat 8.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 7.0.1

This issue was fixed in the openstack/heat 7.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.