Floating IP agent gateway IP addresses not released when deleting dead DVR L3 agents

[Pierre Riteau]

We have a system with some dead DVR hypervisors. Ports of type network:floatingip_agent_gateway are still associated with them and in ACTIVE state. Deleting the L3 agent doesn't delete the ports, which means extra floating IPs are still consumed.

* Version

OpenStack Train deployed with Kolla-Ansible
CentOS 7 binary containers on CentOS 7 hosts
RPM: openstack-neutron-15.0.2-1.el7.noarch

* Environment: Neutron ML2 OVS + DVR

[LIU Yulong (dragon889)]

IMO, you could delete those stale ports manually.

[Pierre Riteau (priteau)]

That's what I did of course, but IMHO Neutron should do it.

[Dan Radez (dradez)]

This appears to be related to:
https://src.fedoraproject.org/rpms/dnsmasq/c/744ba31be775c11b1f52104d6285509b06b81035?branch=master

I'm cleaning my env to do a final verification. Seems that dnsmasq will only listen on lo with this in place.

Maybe we can add interface= to the neutron dnsmasq command to ensure that we always listen on all interfaces no matter what the system config states?

[Pierre Riteau (priteau)]

@Dan: Is your comment for this bug? I don't see the link between dnsmasq and this issue.

[Dan Radez (dradez)]

@Pierre, nope, sry about that.
Commented on the wrong bug :(

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/755330

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/787691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/787691
Committed: https://opendev.org/openstack/neutron/commit/8cc7c0cf7a5196103b097fae67eccbd5dc3980ac
Submitter: "Zuul (22348)"
Branch: master

commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/797757

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/797758

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/797759

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/797760

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/797807

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/797761

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/797762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797757
Committed: https://opendev.org/openstack/neutron/commit/170e6306e20399f60815d7d6e87983de004b83e9
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 170e6306e20399f60815d7d6e87983de004b83e9
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797760
Committed: https://opendev.org/openstack/neutron/commit/4bdfc711453e9b6af955be62a28ef9fcc46a5040
Submitter: "Zuul (22348)"
Branch: stable/train

commit 4bdfc711453e9b6af955be62a28ef9fcc46a5040
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797758
Committed: https://opendev.org/openstack/neutron/commit/dcf050951d71a3c2f3e3d997c6b171fa4483bc20
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit dcf050951d71a3c2f3e3d997c6b171fa4483bc20
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797759
Committed: https://opendev.org/openstack/neutron/commit/96259e96c76475f11ec91d41749d1aca56598427
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 96259e96c76475f11ec91d41749d1aca56598427
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797807
Committed: https://opendev.org/openstack/neutron/commit/ef476bb875e64164a046f3787ca7ca77aaef2bcf
Submitter: "Zuul (22348)"
Branch: stable/stein

commit ef476bb875e64164a046f3787ca7ca77aaef2bcf
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Conflicts:
        neutron/tests/unit/db/test_l3_dvr_db.py

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797762
Committed: https://opendev.org/openstack/neutron/commit/5d932209f6e55ea8d7f76c359559d9d2c8091a5e
Submitter: "Zuul (22348)"
Branch: stable/queens

commit 5d932209f6e55ea8d7f76c359559d9d2c8091a5e
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Conflicts:
        neutron/tests/unit/db/test_l3_dvr_db.py

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/797761
Committed: https://opendev.org/openstack/neutron/commit/8eaa61e916da62d4246a964e7bbcb81b9d5a8506
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 8eaa61e916da62d4246a964e7bbcb81b9d5a8506
Author: Slawek Kaplonski <email address hidden>
Date: Fri Apr 23 10:45:14 2021 +0200

    Remove FIP agent's gw port when L3 agent is deleted

    Floating IP agent gateway ports are created for each external network
    for each node where DVR L3 agent is running and where there is some FIP
    from the ext_net.
    But even, if L3 agent is removed (e.g. when scaling down the cluster),
    such floating IP gateway port is never removed so it consumes IP address
    from the external network.

    With this patch when the DVR L3 agent is deleted, all such fip gateway
    ports owned by that agent will be deleted.
    When new L3 agent is created (registered in the DB), Neutron will check
    if there are any floating IPs on that host and will recreate such FIP
    gateway ports for it.

    Conflicts:
        neutron/tests/unit/db/test_l3_dvr_db.py

    Closes-Bug: #1891360
    Change-Id: If6ef990baf039c556d7420962ac4c54608711f06
    (cherry picked from commit 8cc7c0cf7a5196103b097fae67eccbd5dc3980ac)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.