qrouter ns ip rules not deleted when fip removed from vm

[Edward Hope-Morley]

[Impact]

neutron-l3-agent restart causes partial loss of fip information such that fip removal from vm results in ip rules left behind which breaks external network access for that vm.

[Test Case]

* deploy openstack with dvr enabled
* create distributed router, network etc
* create a vm and attach a floating ip
* go to compute host on which vm is running and restart neutron-l3-agent
* tail -f /var/log/neutron/neutron-l3-agent.log until it settles
* remove fip from vm
* run https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b on that compute node
* should return with "nothing to do"

[Regression Potential]

the patch is reloading, on agent startup, information associated with floating ips, specifically the information needed to delete ip rules and rule priorities associated with a floating ip. Since that is essentially read-only I don't envisage a regression potential. When the l3-agent comes to use that information to delete the floating ip an error could occur if the information it is trying to delete no longer exists but that would not be a problem introduced by this patch so again, I don't envisage any potential for regressions from this patch since it doesn't change behavior in any way other than allowing the l3-agent to behave the same as if it hadn't been restarted.

[Other Info]
patched neutron l3 agent will reload info for *used* floating ips when restarted BUT if there are ip rules left behind from fips removed prior to using a pathed neutron then manual cleanup is still required and for that you can use https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b.

--------------------------------------------------------------------------

With Bionic Stein using dvr_snat if I add a floating ip to a vm then remove the floating ip, the corresponding ip rules in the associated qrouter ns local to the instance are not deleted which results in no longer being able to reach the external network because packets are still sent to the fip namespace (via rfp-/fpr-) e.g. in my compute host running a vm whose address is 192.168.21.28 for which i have removed the fip I still see:

# ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip rule list
0: from all lookup local
32765: from 192.168.21.28 lookup 16
32766: from all lookup main
32767: from all lookup default
3232240897: from 192.168.21.1/24 lookup 3232240897
3232241231: from 192.168.22.79/24 lookup 3232241231

And table 16 leads to:

# ip netns exec qrouter-5e45608f-33d4-41bf-b3ba-915adf612e65 ip route show table 16
default via 169.254.109.249 dev rfp-5e45608f-3

Which results in the instance no longer being able to reach the external network (packets are never sent to the snat- ns in my case).

The workaround is to delete that ip rule but neutron should be taking care of this. Looks like the culprit is in neutron/agent/l3/dvr_local_router.py:floating_ip_removed_dist

Note that the NAT rules were successfully removed from iptables so looks like it is just this bit that is left behind.

[Edward Hope-Morley (hopem)]

ok a bit more info, this problem only occurs if you restart the neutron-l3-agent in between adding and removing the floating ip. Looking at the code it looks like this is because the information needed to delete the fip is held in memory and comes from when the fip is added at which point it is added to floating_ips_dict which is never repopulated. So basically if you restart your l3-agent you lost all records of floating ips that need their ip rules deleted.

    def _add_floating_ip_rule(self, floating_ip, fixed_ip):
        rule_pr = self.fip_ns.allocate_rule_priority(floating_ip)
        self.floating_ips_dict[floating_ip] = (fixed_ip, rule_pr)

        ip_lib.add_ip_rule(namespace=self.ns_name, ip=fixed_ip,
                           table=dvr_fip_ns.FIP_RT_TBL,
                           priority=int(str(rule_pr)))

    def _remove_floating_ip_rule(self, floating_ip):
        if floating_ip in self.floating_ips_dict:
            fixed_ip, rule_pr = self.floating_ips_dict[floating_ip]
            ip_lib.delete_ip_rule(self.ns_name, ip=fixed_ip,
                                  table=dvr_fip_ns.FIP_RT_TBL,
                                  priority=int(str(rule_pr)))
            self.fip_ns.deallocate_rule_priority(floating_ip)
            # TODO(rajeev): Handle else case - exception/log?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/746336

[Edward Hope-Morley (hopem)]

I have put a workaround for this issue here - https://gist.github.com/dosaboy/eca8dcd4560f68d856f465ca8382c58b

Note that this workaround is required even after the fix lands as the l3 agent will not be able to delete old allocations left behind by this bug.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746336
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5eca44bfa850e6e75c9974ae7711b87764628253
Submitter: Zuul
Branch: master

commit 5eca44bfa850e6e75c9974ae7711b87764628253
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673

[Edward Hope-Morley (hopem)]

Stable backports submitted upstream: https://review.opendev.org/#/q/topic:bug/1891673+(status:open+OR+status:merged)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1891673-focal-ussuri.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Corey Bryant (corey.bryant)]

Thanks Ed, this has been uploaded to the corresponding unapproved queues and cloud archive staging PPAs.

[Robie Basak (racb)]

For the Ubuntu SRU:

Please fix this in Groovy first, or explain why an exception to policy is warranted in this specific case.

> [Regression Potential]
> none expected

This is not acceptable. Please see https://wiki.ubuntu.com/StableReleaseUpdates#Procedure and provide the analysis required.

[Edward Hope-Morley (hopem)]

@racb done.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb
Submitter: Zuul
Branch: stable/ussuri

commit 8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673
    (cherry picked from commit 5eca44bfa850e6e75c9974ae7711b87764628253)

[Edward Hope-Morley (hopem)]

The build in groovy-updates [1] matches that in victoria-staging [2] and both have the fix hence why I am marking them as Fix Commited (since groovy/victoria is not released yet).

[1] https://launchpad.net/ubuntu/+source/neutron/2:17.0.0~git2020091014.215a541bd4-0ubuntu1
[2] https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/victoria-staging/+build/19932691

[Brian Murray (brian-murray)]

Hello Edward, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Edward, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Edward, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Edward Hope-Morley (hopem)]

@brian-murray the focal-proposed build appears to have failed as a result of a timeout. I've requested a rebuild on #ubuntu-server so hopefully it will pass next time. I will move on to verifying the others and circle back to focal once the build succeeds.

[Edward Hope-Morley (hopem)]

bionic-ussuri verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:16.1.0-0ubuntu2~cloud0
  Candidate: 2:16.1.0-0ubuntu2~cloud0
  Version table:
 *** 2:16.1.0-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

focal/ussuri verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:16.1.0-0ubuntu2
  Candidate: 2:16.1.0-0ubuntu2
  Version table:
 *** 2:16.1.0-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:16.1.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     2:16.0.0~b3~git2020041516.5f42488a9a-0ubuntu2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

bionic-train verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:15.1.0-0ubuntu1~cloud1
  Candidate: 2:15.1.0-0ubuntu1~cloud1
  Version table:
 *** 2:15.1.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

bionic-stein verified with [Test Case] and output is:

# apt-cache policy neutron-l3-agent
neutron-l3-agent:
  Installed: 2:14.3.0-0ubuntu1~cloud2
  Candidate: 2:14.3.0-0ubuntu1~cloud2
  Version table:
 *** 2:14.3.0-0ubuntu1~cloud2 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

bionic-rocky verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:13.0.7-0ubuntu1~cloud2
  Candidate: 2:13.0.7-0ubuntu1~cloud2
  Version table:
 *** 2:13.0.7-0ubuntu1~cloud2 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

bionic-queens verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:12.1.1-0ubuntu2
  Candidate: 2:12.1.1-0ubuntu2
  Version table:
 *** 2:12.1.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Edward Hope-Morley (hopem)]

xenial-queens verified with [Test Case] and output is:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:12.1.1-0ubuntu2~cloud0
  Candidate: 2:12.1.1-0ubuntu2~cloud0
  Version table:
 *** 2:12.1.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/queens/main amd64 Packages
        100 /var/lib/dpkg/status
     2:8.4.0-0ubuntu7.5 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     2:8.4.0-0ubuntu7.4 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     2:8.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

# ./discover_ip_rules_affected_by_lp1891673.sh
Nothing to cleanup - exiting.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:16.1.0-0ubuntu2

---------------
neutron (2:16.1.0-0ubuntu2) focal; urgency=medium

  * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
    Backport fix for dvr fip ip rule cleanup (LP: #1891673)

 -- Edward Hope-Morley <email address hidden> Wed, 09 Sep 2020 09:25:20 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu2

---------------
neutron (2:12.1.1-0ubuntu2) bionic; urgency=medium

  * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
    Backport fix for dvr fip ip rule cleanup (LP: #1891673)

 -- Edward Hope-Morley <email address hidden> Tue, 08 Sep 2020 17:55:17 +0100

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:16.1.0-0ubuntu2~cloud0
---------------

 neutron (2:16.1.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:16.1.0-0ubuntu2) focal; urgency=medium
 .
   * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
     Backport fix for dvr fip ip rule cleanup (LP: #1891673)

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:15.1.0-0ubuntu1~cloud1
---------------

 neutron (2:15.1.0-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
     Backport fix for dvr fip ip rule cleanup (LP: #1891673)

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:14.3.0-0ubuntu1~cloud2
---------------

 neutron (2:14.3.0-0ubuntu1~cloud2) bionic-stein; urgency=medium
 .
   * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
     Backport fix for dvr fip ip rule cleanup (LP: #1891673)

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:13.0.7-0ubuntu1~cloud2
---------------

 neutron (2:13.0.7-0ubuntu1~cloud2) bionic-rocky; urgency=medium
 .
   * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
     Backport fix for dvr fip ip rule cleanup (LP: #1891673)

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu2~cloud0
---------------

 neutron (2:12.1.1-0ubuntu2~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.1.1-0ubuntu2) bionic; urgency=medium
 .
   * d/p/Ensure-fip-ip-rules-deleted-when-fip-removed.patch
     Backport fix for dvr fip ip rule cleanup (LP: #1891673)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750396
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f28788f77798e6e1e64ac9f60a82b99b52546f8f
Submitter: Zuul
Branch: stable/stein

commit f28788f77798e6e1e64ac9f60a82b99b52546f8f
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673
    (cherry picked from commit 5eca44bfa850e6e75c9974ae7711b87764628253)
    (cherry picked from commit 8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb)
    (cherry picked from commit 84d38f342bcad6537971d732a4961334a5890f3b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f63e3595c4e2aede7168fb1e65dbe7190308adce
Submitter: Zuul
Branch: stable/queens

commit f63e3595c4e2aede7168fb1e65dbe7190308adce
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673
    (cherry picked from commit 5eca44bfa850e6e75c9974ae7711b87764628253)
    (cherry picked from commit 8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb)
    (cherry picked from commit 84d38f342bcad6537971d732a4961334a5890f3b)
    (cherry picked from commit f28788f77798e6e1e64ac9f60a82b99b52546f8f)
    (cherry picked from commit 1eb5b54776d2194319528712399439c54c5320d7)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1eb5b54776d2194319528712399439c54c5320d7
Submitter: Zuul
Branch: stable/rocky

commit 1eb5b54776d2194319528712399439c54c5320d7
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673
    (cherry picked from commit 5eca44bfa850e6e75c9974ae7711b87764628253)
    (cherry picked from commit 8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb)
    (cherry picked from commit 84d38f342bcad6537971d732a4961334a5890f3b)
    (cherry picked from commit f28788f77798e6e1e64ac9f60a82b99b52546f8f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/750395
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=84d38f342bcad6537971d732a4961334a5890f3b
Submitter: Zuul
Branch: stable/train

commit 84d38f342bcad6537971d732a4961334a5890f3b
Author: Edward Hope-Morley <email address hidden>
Date: Fri Aug 14 17:44:54 2020 +0100

    Ensure fip ip rules deleted when fip removed

    The information needed to delete ip rules associated
    with fips is held in memory between add and remove so
    a restart of the l3-agent results in any fips that
    existed before the restart having their ip rules
    persist after the fips are removed. This patch
    enures that an agent restart reloads this information
    so that ip rules associated with a fip are correctly
    removed when the fip is removed.

    Change-Id: If656a703c996ccc7719b1b09d793c5bbdfd6f3c1
    Closes-Bug: #1891673
    (cherry picked from commit 5eca44bfa850e6e75c9974ae7711b87764628253)
    (cherry picked from commit 8ba796ea7ff28a815996ffeaf3c4dc39df1edcfb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/794604

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/794604
Committed: https://opendev.org/openstack/neutron/commit/a03c240ef4ea1d4b874b618dbd0163a3a2f7024c
Submitter: "Zuul (22348)"
Branch: master

commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/810393

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/810394

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/810425

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/810395

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/810396

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/810427

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/810397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810425
Committed: https://opendev.org/openstack/neutron/commit/7eaa84a0cd48adb2ecd7653640d32aea8096e786
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7eaa84a0cd48adb2ecd7653640d32aea8096e786
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810393
Committed: https://opendev.org/openstack/neutron/commit/f2f7602de7e1b067ae28f55b2b2dbbde29e7bcb1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f2f7602de7e1b067ae28f55b2b2dbbde29e7bcb1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810394
Committed: https://opendev.org/openstack/neutron/commit/b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810395
Committed: https://opendev.org/openstack/neutron/commit/2cde9609c972e4fc4dd11a5c4caa342a3c1a3272
Submitter: "Zuul (22348)"
Branch: stable/train

commit 2cde9609c972e4fc4dd11a5c4caa342a3c1a3272
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810427
Committed: https://opendev.org/openstack/neutron/commit/d87f4f11c7aec16a68efec17eb889abe3b1f9ee2
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit d87f4f11c7aec16a68efec17eb889abe3b1f9ee2
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/agent/l3/dvr_local_router.py
        neutron/tests/unit/agent/l3/test_agent.py
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    That patch additionally changes to use old IpRuleCommand.add() method
    from the ip_lib module as in that branch it's not yet moved to use
    pyroute2 library. Additional changes are in the
    neutron/agent/l3/dvr_local_router.py file.

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)
    (cherry picked from commit 34ec42df9fb2c1ee662928ee14be8986acd9f15b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810396
Committed: https://opendev.org/openstack/neutron/commit/d863a1cb7c88caf1eb63aa1c06c19e41aea7c581
Submitter: "Zuul (22348)"
Branch: stable/stein

commit d863a1cb7c88caf1eb63aa1c06c19e41aea7c581
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/810397
Committed: https://opendev.org/openstack/neutron/commit/b950fff6700bdfaacbf64415b322525ddcab85ef
Submitter: "Zuul (22348)"
Branch: stable/queens

commit b950fff6700bdfaacbf64415b322525ddcab85ef
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Jun 3 14:49:45 2021 +0000

    Populate self.floating_ips_dict using "ip rule" information

    When the L3 agent starts, reads the floating IP rule priority from
    a state file created by "FipRulePriorityAllocator". In case of not
    having all floating IPs registers in this file, the method:
    - Creates a new priority for this floating IP.
    - Creates the "ip rule" in the namespace.
    - Adds a new entry in "self.floating_ips_dict".

    All "ip rules" present in the namespace that do not match the
    registered fixed IP address ("from") and the priority assigned
    are deleted.

    Closes-Bug: #1891673
    Closes-Bug: #1929821

    Conflicts:
        neutron/agent/l3/dvr_local_router.py
        neutron/tests/unit/agent/l3/test_agent.py
        neutron/tests/unit/agent/l3/test_dvr_local_router.py

    That patch additionally changes to use old IpRuleCommand.add() method
    from the ip_lib module as in that branch it's not yet moved to use
    pyroute2 library. Additional changes are in the
    neutron/agent/l3/dvr_local_router.py file.

    Change-Id: Ia3fbde3304ab5f3c309dc62dbf58274afbcf4614
    (cherry picked from commit a03c240ef4ea1d4b874b618dbd0163a3a2f7024c)
    (cherry picked from commit b4ad1a2775d00cd6d14bd4766a0a1c5c41332d89)
    (cherry picked from commit 34ec42df9fb2c1ee662928ee14be8986acd9f15b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.