[dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host

[ITD27M01]

[Impact]
When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening.

[Test Case]
 * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts.
 * ensure min. 2 compute hosts
 * create one ext network and one private network
 * add private subnet to router and ext as gateway
 * check which compute has the snat ns (ip netns| grep snat)
 * create a vm on each compute host
 * check that qrouter ns on both computes has rfp interface
 * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp
 * disable and re-enable router
 * openstack router set --disable <router>; openstack router set --enable <router>
 * check again
 * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp

[Where problems could occur]
no regression is expected, but if one occurs it would likely result in breakage with external network connectivity

-------------------------------------------------------------------------

Hello,

In the case of dvr_snat l3 agents are deployed on hypervisors there can be race condition. The agent creates snat namespaces on each scheduled host and removes them at second step. At this second step agent removes the rfp interface from qrouter even when there is VM with floating IP on the host.

When VM is deployed at the time of second step we can lost external access to VMs floating IP. The issue can be reproduced by hand:

1. Create tenant network and router with external gateway
2. Create VM with floating ip
3. Ensure that VM on the hypervisor without snat-* namespace
4. Set the router to disabled state (openstack router set --disable <router>)
5. Set the router to enabled state (openstack router set --enabled <router>)
6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface.

Environment:

1. Neutron with ML2 OVS plugin.
2. L3 agents in dvr_snat mode on each hypervisor
3. openstack-neutron-common-15.1.1-0.20200611111910.7d97420.el8ost.noarch

[Edward Hope-Morley (hopem)]

Patch: https://review.opendev.org/c/openstack/neutron/+/775372

[Corey Bryant (corey.bryant)]

Ed, thanks for the patches. One comment to make this easier for the SRU team to review. Would you be able to switch the [Regression Potential] section to [Where problems could occur] and update the contents accordingly? The SRU team recently renamed this in order to get an analysis of what a regression could look like in the changed code.

[Corey Bryant (corey.bryant)]

New versions of neutron with this fix have been uploaded to hirsute, and groovy/focal unapproved queues.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:17.1.0+git2021012815.0fb63f7297-0ubuntu3

---------------
neutron (2:17.1.0+git2021012815.0fb63f7297-0ubuntu3) hirsute; urgency=medium

  [ Edward Hope-Morley ]
  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

 -- Corey Bryant <email address hidden> Wed, 24 Feb 2021 12:41:20 -0500

[Łukasz Zemczak (sil2100)]

An upload of neutron to groovy-proposed has been rejected from the upload queue for the following reason: "The .changes file is missing the 2:17.1.0-0ubuntu1 changelog entry, so the LB bugs field includes only one bug (without the new-upstream version tracking bug). Please re-build source with -v2:17.0.0-0ubuntu3 and re-upload.".

[Łukasz Zemczak (sil2100)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:17.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.3.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:17.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.3.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Trent Lloyd (lathiat)]

When using DVR-SNAT, a simple neutron-l3-agent gateway restart triggers this issue.

Reproduction Note: Nodes with an ACTIVE or BACKUP (in the case of L3HA) router for the network are not affected by this issue, so a small 1-6 node environment may make this difficult to reproduce or only affect half of the nodes (e.g. 3/6 nodes if you have L3HA).

Workaround: for each compute node, you need to create a new VM on each network. While registering the new VM port it will cause the missing fpr/rfp interface pair to be created and paired. It does not seem possible to fix it any other way such as stopping/starting the existing VM, rebooting the host, etc.

[Hemanth Nakkina (hemanth-n)]

Verified on groovy-proposed and fix is working fine as per the testcase

Ping to floating ip failed when router is disabled and succeed a few milliseconds after the router is enabled.

$ ping 10.5.150.53
PING 10.5.150.53 (10.5.150.53) 56(84) bytes of data.
64 bytes from 10.5.150.53: icmp_seq=1 ttl=62 time=0.788 ms
64 bytes from 10.5.150.53: icmp_seq=2 ttl=62 time=0.809 ms
64 bytes from 10.5.150.53: icmp_seq=3 ttl=62 time=1.07 ms
64 bytes from 10.5.150.53: icmp_seq=4 ttl=62 time=0.740 ms
64 bytes from 10.5.150.53: icmp_seq=5 ttl=62 time=0.919 ms
64 bytes from 10.5.150.53: icmp_seq=6 ttl=62 time=0.893 ms
64 bytes from 10.5.150.53: icmp_seq=7 ttl=62 time=0.901 ms
64 bytes from 10.5.150.53: icmp_seq=8 ttl=62 time=0.838 ms
From 10.5.153.226 icmp_seq=11 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=12 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=13 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=14 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=15 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=16 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=18 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=19 Destination Host Unreachable
From 10.5.153.226 icmp_seq=20 Destination Host Unreachable
From 10.5.0.5 icmp_seq=21 Destination Host Unreachable
From 10.5.0.5 icmp_seq=22 Destination Host Unreachable
From 10.5.0.5 icmp_seq=23 Destination Host Unreachable
From 10.5.0.5 icmp_seq=24 Destination Host Unreachable
From 10.5.0.5 icmp_seq=25 Destination Host Unreachable
From 10.5.0.5 icmp_seq=26 Destination Host Unreachable
From 10.5.0.5 icmp_seq=27 Destination Host Unreachable
From 10.5.0.5 icmp_seq=28 Destination Host Unreachable
From 10.5.0.5 icmp_seq=29 Destination Host Unreachable
From 10.5.0.5 icmp_seq=30 Destination Host Unreachable
From 10.5.0.5 icmp_seq=31 Destination Host Unreachable
From 10.5.0.5 icmp_seq=32 Destination Host Unreachable
From 10.5.0.5 icmp_seq=33 Destination Host Unreachable
64 bytes from 10.5.150.53: icmp_seq=34 ttl=62 time=2248 ms
64 bytes from 10.5.150.53: icmp_seq=36 ttl=62 time=200 ms
64 bytes from 10.5.150.53: icmp_seq=35 ttl=62 time=1224 ms

[Hemanth Nakkina (hemanth-n)]

Verified on focal-proposed and fix is working fine as per the testcase

Ping to floating ip failed when router is disabled and succeed a few milliseconds after the router is enabled.

$ ping 10.5.151.34
PING 10.5.151.34 (10.5.151.34) 56(84) bytes of data.
64 bytes from 10.5.151.34: icmp_seq=1 ttl=62 time=3.19 ms
64 bytes from 10.5.151.34: icmp_seq=2 ttl=62 time=0.990 ms
64 bytes from 10.5.151.34: icmp_seq=3 ttl=62 time=1.18 ms
64 bytes from 10.5.151.34: icmp_seq=4 ttl=62 time=0.893 ms
64 bytes from 10.5.151.34: icmp_seq=5 ttl=62 time=0.876 ms
64 bytes from 10.5.151.34: icmp_seq=6 ttl=62 time=0.890 ms
64 bytes from 10.5.151.34: icmp_seq=7 ttl=62 time=0.804 ms
64 bytes from 10.5.151.34: icmp_seq=8 ttl=62 time=0.705 ms
64 bytes from 10.5.151.34: icmp_seq=9 ttl=62 time=0.877 ms
From 10.5.151.236 icmp_seq=11 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=12 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=13 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=11 Destination Host Unreachable
From 10.5.151.236 icmp_seq=15 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=16 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=14 Destination Host Unreachable
From 10.5.151.236 icmp_seq=18 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=20 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=23 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=21 Destination Host Unreachable
From 10.5.151.236 icmp_seq=24 Destination Host Unreachable
From 10.5.151.236 icmp_seq=25 Destination Host Unreachable
From 10.5.151.236 icmp_seq=26 Destination Host Unreachable
From 10.5.151.236 icmp_seq=28 Destination Host Unreachable
From 10.5.151.236 icmp_seq=29 Destination Host Unreachable
From 10.5.151.236 icmp_seq=30 Destination Host Unreachable
64 bytes from 10.5.151.34: icmp_seq=31 ttl=62 time=1.02 ms
64 bytes from 10.5.151.34: icmp_seq=32 ttl=62 time=0.821 ms
64 bytes from 10.5.151.34: icmp_seq=33 ttl=62 time=0.738 ms
64 bytes from 10.5.151.34: icmp_seq=34 ttl=62 time=0.670 ms

[Hemanth Nakkina (hemanth-n)]

Verified focal-victoria and bionic-ussuri and fix is working fine as per the testcase.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.1 release.

[Paul Goins (vultaire)]

We are encountering this on bionic/stein; is a backport in the works?

[Hemanth Nakkina (hemanth-n)]

debdiff for UCA train

[Hemanth Nakkina (hemanth-n)]

debdiff for UCA stein

[Hemanth Nakkina (hemanth-n)]

@Paul Goins, I submitted debdiff's for UCA train and stein. Will wait for upstream rocky/queens to get merged to submit debdiffs.

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:16.3.0-0ubuntu3

---------------
neutron (2:16.3.0-0ubuntu3) focal; urgency=medium

  * d/p/revert-dvr-remove-control-plane-arp-updates.patch: Cherry-picked
    from https://review.opendev.org/c/openstack/neutron/+/777903 to prevent
    permanent arp entries that never get deleted (LP: #1916761).
  * d/p/improve-get-devices-with-ip-performance.patch: Performance of
    get_devices_with_ip is improved to limit the amount of information
    to be sent and reduce the number of syscalls. (LP: #1896734).

neutron (2:16.3.0-0ubuntu2) focal; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

neutron (2:16.3.0-0ubuntu1) focal; urgency=medium

  * d/watch: Add trailing slash to Neutron URL.
  * New stable point release for OpenStack Ussuri (LP: #1915786).
  * d/p/fix-removal-of-dvr-src-mac-flows.patch,
    d/p/ovn-fix-inconsistent-igmp-configuration.patch: Removed after patch landed
    upstream.

 -- Corey Bryant <email address hidden> Mon, 08 Mar 2021 13:26:42 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:17.1.0-0ubuntu3

---------------
neutron (2:17.1.0-0ubuntu3) groovy; urgency=medium

  * d/p/revert-dvr-remove-control-plane-arp-updates.patch: Cherry-picked
    from https://review.opendev.org/c/openstack/neutron/+/777903 to prevent
    permanent arp entries that never get deleted (LP: #1916761).
  * d/p/improve-get-devices-with-ip-performance.patch: Performance of
    get_devices_with_ip is improved to limit the amount of information
    to be sent and reduce the number of syscalls. (LP: #1896734).

neutron (2:17.1.0-0ubuntu2) groovy; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

neutron (2:17.1.0-0ubuntu1) groovy; urgency=medium

  * d/watch: Fix typo in watch URL, add trailing slash.
  * New stable point release for OpenStack Victoria (LP: #1915785).
  * d/p/fix-removal-of-dvr-src-mac-flows.patch,
    d/p/ovn-fix-inconsistent-igmp-configuration.patch: Removed after fix landed
    upstream.

 -- Corey Bryant <email address hidden> Mon, 08 Mar 2021 13:00:21 -0500

[Hemanth Nakkina (hemanth-n)]

Debdiff for UCA rocky

[Hemanth Nakkina (hemanth-n)]

Debdiff for bionic

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[Chris MacNaughton (chris.macnaughton)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was released into the Victoria cloud archive but a hiccup in the automatic backport process from distro caused this change notification to get lost.

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

@Hemanth, there are some unit tests that fail for bionic with the debdiff applied. Can you take a look? https://paste.ubuntu.com/p/Ky98vGbkrQ/

[Hemanth Nakkina (hemanth-n)]

Verified the test case on bionic-train, bionic-stein and bionic-rocky and the test case is successful. The connectivity to the instance is restored after the router is enabled.

[Hemanth Nakkina (hemanth-n)]

@Corey the following patch (from upstream stable/queens) is not present in Ubuntu neutron package 2:12.1.1-0ubuntu4~cloud0
https://review.opendev.org/c/openstack/neutron/+/709406
This resulted in failure of few unit tests on ubuntu neutron package.

The unittests can also be resolved by adding the below line after https://opendev.org/openstack/neutron/src/branch/stable/queens/neutron/agent/l3/dvr_edge_router.py#L188

--- a/neutron/agent/l3/dvr_edge_router.py
+++ b/neutron/agent/l3/dvr_edge_router.py
@@ -186,6 +186,7 @@ class DvrEdgeRouter(dvr_local_router.DvrLocalRouter):
         # is to move the creation of the snat namespace here
         if self._is_this_snat_host():
             self.snat_namespace.create()
+ return self.snat_namespace

Please suggest if we need to SRU https://review.opendev.org/c/openstack/neutron/+/709406 in ubuntu neutron package or just update the debdiff with single line of code change

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:15.3.2-0ubuntu1~cloud2
---------------

 neutron (2:15.3.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:14.4.2-0ubuntu1~cloud2
---------------

 neutron (2:14.4.2-0ubuntu1~cloud2) bionic-stein; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:13.0.7-0ubuntu1~cloud5
---------------

 neutron (2:13.0.7-0ubuntu1~cloud5) bionic-rocky; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

[Hemanth Nakkina (hemanth-n)]

Unit tests for Bionic patch are failing due to missing https://review.opendev.org/c/openstack/neutron/+/709406.

So I submitted a SRU for corresponding bug -
https://bugs.launchpad.net/neutron/+bug/1850779

[Chris MacNaughton (chris.macnaughton)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Hemanth Nakkina (hemanth-n)]

Verified on bionic-train and working fine.

Output of ping VM FIP (launched on non-snat node) during disable/enable router

$ ping 10.5.150.244
PING 10.5.150.244 (10.5.150.244) 56(84) bytes of data.
64 bytes from 10.5.150.244: icmp_seq=1 ttl=62 time=393 ms
64 bytes from 10.5.150.244: icmp_seq=2 ttl=62 time=0.955 ms
From 10.5.151.36 icmp_seq=8 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=9 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=10 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=17 Destination Host Unreachable
From 10.5.151.36 icmp_seq=20 Destination Host Unreachable
From 10.5.151.36 icmp_seq=21 Destination Host Unreachable
From 10.5.151.36 icmp_seq=22 Destination Host Unreachable
64 bytes from 10.5.150.244: icmp_seq=27 ttl=62 time=1.10 ms
64 bytes from 10.5.150.244: icmp_seq=28 ttl=62 time=0.750 ms
64 bytes from 10.5.150.244: icmp_seq=29 ttl=62 time=0.840 ms

[Robie Basak (racb)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello ITD27M01, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Hemanth Nakkina (hemanth-n)]

The test case is verified on bionic-proposed and the external access to VM is restored once the router is re-enabled.

[Hemanth Nakkina (hemanth-n)]

The test case is verified on xenial-queens with queens-proposed packages and the external access to VM is restored once the router is re-enabled.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:15.3.3-0ubuntu1~cloud0
---------------

 neutron (2:15.3.3-0ubuntu1~cloud0) bionic-train; urgency=medium
 .
   * New stable point release for OpenStack Train (LP: #1923038).
   * d/p/revert-dvr-remove-control-plane-arp-updates.patch, d/p/0001-Fix-
     deletion-of-rfp-interfaces-when-router-is-re-ena.patch: Removed
     after fix landed upstream.
 .
 neutron (2:15.3.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7

---------------
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium

  * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
    - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
      (LP: #1849098).

neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium

  * Do not initialize snat-ns twice (LP: #1850779)
    - d/p/0001-Do-not-initialize-snat-ns-twice.patch

neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

 -- Seyeong Kim <email address hidden> Mon, 03 May 2021 17:15:28 +0900

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7~cloud0
---------------

 neutron (2:12.1.1-0ubuntu7~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium
 .
   * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
     - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
       (LP: #1849098).
 .
 neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium
 .
   * Do not initialize snat-ns twice (LP: #1850779)
     - d/p/0001-Do-not-initialize-snat-ns-twice.patch
 .
 neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.