[dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host

Bug #1894843 reported by ITD27M01
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Medium
Unassigned
Queens
High
Unassigned
Rocky
High
Unassigned
Stein
High
Unassigned
Train
High
Unassigned
Ussuri
Medium
Unassigned
Victoria
Medium
Unassigned
neutron
Medium
Hemanth Nakkina
neutron (Ubuntu)
Medium
Unassigned
Bionic
High
Unassigned
Focal
Medium
Unassigned
Groovy
Medium
Unassigned
Hirsute
Medium
Unassigned

Bug Description

[Impact]
When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening.

[Test Case]
 * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts.
 * ensure min. 2 compute hosts
 * create one ext network and one private network
 * add private subnet to router and ext as gateway
 * check which compute has the snat ns (ip netns| grep snat)
 * create a vm on each compute host
 * check that qrouter ns on both computes has rfp interface
 * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp
 * disable and re-enable router
 * openstack router set --disable <router>; openstack router set --enable <router>
 * check again
 * ip netns| grep qrouter; ip netns exec <ns> ip a s| grep rfp

[Where problems could occur]
no regression is expected, but if one occurs it would likely result in breakage with external network connectivity

-------------------------------------------------------------------------

Hello,

In the case of dvr_snat l3 agents are deployed on hypervisors there can be race condition. The agent creates snat namespaces on each scheduled host and removes them at second step. At this second step agent removes the rfp interface from qrouter even when there is VM with floating IP on the host.

When VM is deployed at the time of second step we can lost external access to VMs floating IP. The issue can be reproduced by hand:

1. Create tenant network and router with external gateway
2. Create VM with floating ip
3. Ensure that VM on the hypervisor without snat-* namespace
4. Set the router to disabled state (openstack router set --disable <router>)
5. Set the router to enabled state (openstack router set --enabled <router>)
6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface.

Environment:

1. Neutron with ML2 OVS plugin.
2. L3 agents in dvr_snat mode on each hypervisor
3. openstack-neutron-common-15.1.1-0.20200611111910.7d97420.el8ost.noarch

ITD27M01 (igortiunov)
summary: - [dvr_snat] Router update deletes rfp interface from qrouter event when
- VM port is present on this host
+ [dvr_snat] Router update deletes rfp interface from qrouter even when VM
+ port is present on this host
tags: added: l3-dvr-backlog
Changed in neutron:
importance: Undecided → Medium
Changed in neutron:
assignee: nobody → Hemanth Nakkina (hemanth-n)
tags: added: seg
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Changed in neutron:
status: New → In Progress
Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Revision history for this message
Edward Hope-Morley (hopem) wrote :
description: updated
Changed in neutron (Ubuntu Hirsute):
status: New → Fix Released
Revision history for this message
Edward Hope-Morley (hopem) wrote :
Changed in neutron (Ubuntu Hirsute):
status: Fix Released → Triaged
importance: Undecided → Medium
Changed in neutron (Ubuntu Groovy):
status: New → Triaged
Changed in neutron (Ubuntu Focal):
status: New → Triaged
importance: Undecided → Medium
Changed in neutron (Ubuntu Groovy):
importance: Undecided → Medium
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Ed, thanks for the patches. One comment to make this easier for the SRU team to review. Would you be able to switch the [Regression Potential] section to [Where problems could occur] and update the contents accordingly? The SRU team recently renamed this in order to get an analysis of what a regression could look like in the changed code.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

New versions of neutron with this fix have been uploaded to hirsute, and groovy/focal unapproved queues.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:17.1.0+git2021012815.0fb63f7297-0ubuntu3

---------------
neutron (2:17.1.0+git2021012815.0fb63f7297-0ubuntu3) hirsute; urgency=medium

  [ Edward Hope-Morley ]
  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

 -- Corey Bryant <email address hidden> Wed, 24 Feb 2021 12:41:20 -0500

Changed in neutron (Ubuntu Hirsute):
status: Triaged → Fix Released
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Proposed package upload rejected

An upload of neutron to groovy-proposed has been rejected from the upload queue for the following reason: "The .changes file is missing the 2:17.1.0-0ubuntu1 changelog entry, so the LB bugs field includes only one bug (without the new-upstream version tracking bug). Please re-build source with -v2:17.0.0-0ubuntu3 and re-upload.".

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello ITD27M01, or anyone else affected,

Accepted neutron into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:17.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Groovy):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-archive:
status: Triaged → Fix Committed
tags: added: verification-victoria-needed
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.3.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:17.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.3.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Trent Lloyd (lathiat) wrote :

When using DVR-SNAT, a simple neutron-l3-agent gateway restart triggers this issue.

Reproduction Note: Nodes with an ACTIVE or BACKUP (in the case of L3HA) router for the network are not affected by this issue, so a small 1-6 node environment may make this difficult to reproduce or only affect half of the nodes (e.g. 3/6 nodes if you have L3HA).

Workaround: for each compute node, you need to create a new VM on each network. While registering the new VM port it will cause the missing fpr/rfp interface pair to be created and paired. It does not seem possible to fix it any other way such as stopping/starting the existing VM, rebooting the host, etc.

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified on groovy-proposed and fix is working fine as per the testcase

Ping to floating ip failed when router is disabled and succeed a few milliseconds after the router is enabled.

$ ping 10.5.150.53
PING 10.5.150.53 (10.5.150.53) 56(84) bytes of data.
64 bytes from 10.5.150.53: icmp_seq=1 ttl=62 time=0.788 ms
64 bytes from 10.5.150.53: icmp_seq=2 ttl=62 time=0.809 ms
64 bytes from 10.5.150.53: icmp_seq=3 ttl=62 time=1.07 ms
64 bytes from 10.5.150.53: icmp_seq=4 ttl=62 time=0.740 ms
64 bytes from 10.5.150.53: icmp_seq=5 ttl=62 time=0.919 ms
64 bytes from 10.5.150.53: icmp_seq=6 ttl=62 time=0.893 ms
64 bytes from 10.5.150.53: icmp_seq=7 ttl=62 time=0.901 ms
64 bytes from 10.5.150.53: icmp_seq=8 ttl=62 time=0.838 ms
From 10.5.153.226 icmp_seq=11 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=12 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=13 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=14 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=15 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=16 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=18 Redirect Host(New nexthop: 53.150.5.10)
From 10.5.153.226 icmp_seq=19 Destination Host Unreachable
From 10.5.153.226 icmp_seq=20 Destination Host Unreachable
From 10.5.0.5 icmp_seq=21 Destination Host Unreachable
From 10.5.0.5 icmp_seq=22 Destination Host Unreachable
From 10.5.0.5 icmp_seq=23 Destination Host Unreachable
From 10.5.0.5 icmp_seq=24 Destination Host Unreachable
From 10.5.0.5 icmp_seq=25 Destination Host Unreachable
From 10.5.0.5 icmp_seq=26 Destination Host Unreachable
From 10.5.0.5 icmp_seq=27 Destination Host Unreachable
From 10.5.0.5 icmp_seq=28 Destination Host Unreachable
From 10.5.0.5 icmp_seq=29 Destination Host Unreachable
From 10.5.0.5 icmp_seq=30 Destination Host Unreachable
From 10.5.0.5 icmp_seq=31 Destination Host Unreachable
From 10.5.0.5 icmp_seq=32 Destination Host Unreachable
From 10.5.0.5 icmp_seq=33 Destination Host Unreachable
64 bytes from 10.5.150.53: icmp_seq=34 ttl=62 time=2248 ms
64 bytes from 10.5.150.53: icmp_seq=36 ttl=62 time=200 ms
64 bytes from 10.5.150.53: icmp_seq=35 ttl=62 time=1224 ms

tags: added: verification-done-groovy
removed: verification-needed-groovy
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified on focal-proposed and fix is working fine as per the testcase

Ping to floating ip failed when router is disabled and succeed a few milliseconds after the router is enabled.

$ ping 10.5.151.34
PING 10.5.151.34 (10.5.151.34) 56(84) bytes of data.
64 bytes from 10.5.151.34: icmp_seq=1 ttl=62 time=3.19 ms
64 bytes from 10.5.151.34: icmp_seq=2 ttl=62 time=0.990 ms
64 bytes from 10.5.151.34: icmp_seq=3 ttl=62 time=1.18 ms
64 bytes from 10.5.151.34: icmp_seq=4 ttl=62 time=0.893 ms
64 bytes from 10.5.151.34: icmp_seq=5 ttl=62 time=0.876 ms
64 bytes from 10.5.151.34: icmp_seq=6 ttl=62 time=0.890 ms
64 bytes from 10.5.151.34: icmp_seq=7 ttl=62 time=0.804 ms
64 bytes from 10.5.151.34: icmp_seq=8 ttl=62 time=0.705 ms
64 bytes from 10.5.151.34: icmp_seq=9 ttl=62 time=0.877 ms
From 10.5.151.236 icmp_seq=11 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=12 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=13 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=11 Destination Host Unreachable
From 10.5.151.236 icmp_seq=15 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=16 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=14 Destination Host Unreachable
From 10.5.151.236 icmp_seq=18 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=20 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=23 Redirect Host(New nexthop: 34.151.5.10)
From 10.5.151.236 icmp_seq=21 Destination Host Unreachable
From 10.5.151.236 icmp_seq=24 Destination Host Unreachable
From 10.5.151.236 icmp_seq=25 Destination Host Unreachable
From 10.5.151.236 icmp_seq=26 Destination Host Unreachable
From 10.5.151.236 icmp_seq=28 Destination Host Unreachable
From 10.5.151.236 icmp_seq=29 Destination Host Unreachable
From 10.5.151.236 icmp_seq=30 Destination Host Unreachable
64 bytes from 10.5.151.34: icmp_seq=31 ttl=62 time=1.02 ms
64 bytes from 10.5.151.34: icmp_seq=32 ttl=62 time=0.821 ms
64 bytes from 10.5.151.34: icmp_seq=33 ttl=62 time=0.738 ms
64 bytes from 10.5.151.34: icmp_seq=34 ttl=62 time=0.670 ms

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified focal-victoria and bionic-ussuri and fix is working fine as per the testcase.

tags: added: verification-ussuri-done verification-victoria-done
removed: verification-ussuri-needed verification-victoria-needed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.3.3

This issue was fixed in the openstack/neutron 15.3.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.3.1

This issue was fixed in the openstack/neutron 16.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.1.1

This issue was fixed in the openstack/neutron 17.1.1 release.

Revision history for this message
Paul Goins (vultaire) wrote :

We are encountering this on bionic/stein; is a backport in the works?

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

debdiff for UCA train

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

debdiff for UCA stein

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

@Paul Goins, I submitted debdiff's for UCA train and stein. Will wait for upstream rocky/queens to get merged to submit debdiffs.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:16.3.0-0ubuntu3

---------------
neutron (2:16.3.0-0ubuntu3) focal; urgency=medium

  * d/p/revert-dvr-remove-control-plane-arp-updates.patch: Cherry-picked
    from https://review.opendev.org/c/openstack/neutron/+/777903 to prevent
    permanent arp entries that never get deleted (LP: #1916761).
  * d/p/improve-get-devices-with-ip-performance.patch: Performance of
    get_devices_with_ip is improved to limit the amount of information
    to be sent and reduce the number of syscalls. (LP: #1896734).

neutron (2:16.3.0-0ubuntu2) focal; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

neutron (2:16.3.0-0ubuntu1) focal; urgency=medium

  * d/watch: Add trailing slash to Neutron URL.
  * New stable point release for OpenStack Ussuri (LP: #1915786).
  * d/p/fix-removal-of-dvr-src-mac-flows.patch,
    d/p/ovn-fix-inconsistent-igmp-configuration.patch: Removed after patch landed
    upstream.

 -- Corey Bryant <email address hidden> Mon, 08 Mar 2021 13:26:42 -0500

Changed in neutron (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:17.1.0-0ubuntu3

---------------
neutron (2:17.1.0-0ubuntu3) groovy; urgency=medium

  * d/p/revert-dvr-remove-control-plane-arp-updates.patch: Cherry-picked
    from https://review.opendev.org/c/openstack/neutron/+/777903 to prevent
    permanent arp entries that never get deleted (LP: #1916761).
  * d/p/improve-get-devices-with-ip-performance.patch: Performance of
    get_devices_with_ip is improved to limit the amount of information
    to be sent and reduce the number of syscalls. (LP: #1896734).

neutron (2:17.1.0-0ubuntu2) groovy; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

neutron (2:17.1.0-0ubuntu1) groovy; urgency=medium

  * d/watch: Fix typo in watch URL, add trailing slash.
  * New stable point release for OpenStack Victoria (LP: #1915785).
  * d/p/fix-removal-of-dvr-src-mac-flows.patch,
    d/p/ovn-fix-inconsistent-igmp-configuration.patch: Removed after fix landed
    upstream.

 -- Corey Bryant <email address hidden> Mon, 08 Mar 2021 13:00:21 -0500

Changed in neutron (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Debdiff for UCA rocky

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Debdiff for bionic

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.0.0.0rc1

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was released into the Victoria cloud archive but a hiccup in the automatic backport process from distro caused this change notification to get lost.

tags: added: sts
Changed in neutron (Ubuntu Bionic):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Corey Bryant (corey.bryant) wrote : Please test proposed package

Hello ITD27M01, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

@Hemanth, there are some unit tests that fail for bionic with the debdiff applied. Can you take a look? https://paste.ubuntu.com/p/Ky98vGbkrQ/

tags: added: verification-rocky-needed verification-stein-needed
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified the test case on bionic-train, bionic-stein and bionic-rocky and the test case is successful. The connectivity to the instance is restored after the router is enabled.

tags: added: verification-rocky-done verification-stein-done verification-train-done
removed: verification-rocky-needed verification-stein-needed verification-train-needed
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

@Corey the following patch (from upstream stable/queens) is not present in Ubuntu neutron package 2:12.1.1-0ubuntu4~cloud0
https://review.opendev.org/c/openstack/neutron/+/709406
This resulted in failure of few unit tests on ubuntu neutron package.

The unittests can also be resolved by adding the below line after https://opendev.org/openstack/neutron/src/branch/stable/queens/neutron/agent/l3/dvr_edge_router.py#L188

--- a/neutron/agent/l3/dvr_edge_router.py
+++ b/neutron/agent/l3/dvr_edge_router.py
@@ -186,6 +186,7 @@ class DvrEdgeRouter(dvr_local_router.DvrLocalRouter):
         # is to move the creation of the snat namespace here
         if self._is_this_snat_host():
             self.snat_namespace.create()
+ return self.snat_namespace

Please suggest if we need to SRU https://review.opendev.org/c/openstack/neutron/+/709406 in ubuntu neutron package or just update the debdiff with single line of code change

Revision history for this message
Corey Bryant (corey.bryant) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:15.3.2-0ubuntu1~cloud2
---------------

 neutron (2:15.3.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:14.4.2-0ubuntu1~cloud2
---------------

 neutron (2:14.4.2-0ubuntu1~cloud2) bionic-stein; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:13.0.7-0ubuntu1~cloud5
---------------

 neutron (2:13.0.7-0ubuntu1~cloud5) bionic-rocky; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Unit tests for Bionic patch are failing due to missing https://review.opendev.org/c/openstack/neutron/+/709406.

So I submitted a SRU for corresponding bug -
https://bugs.launchpad.net/neutron/+bug/1850779

Revision history for this message
Chris MacNaughton (chris.macnaughton) wrote : Please test proposed package

Hello ITD27M01, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
removed: verification-train-done
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified on bionic-train and working fine.

Output of ping VM FIP (launched on non-snat node) during disable/enable router

$ ping 10.5.150.244
PING 10.5.150.244 (10.5.150.244) 56(84) bytes of data.
64 bytes from 10.5.150.244: icmp_seq=1 ttl=62 time=393 ms
64 bytes from 10.5.150.244: icmp_seq=2 ttl=62 time=0.955 ms
From 10.5.151.36 icmp_seq=8 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=9 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=10 Redirect Host(New nexthop: 244.150.5.10)
From 10.5.151.36 icmp_seq=17 Destination Host Unreachable
From 10.5.151.36 icmp_seq=20 Destination Host Unreachable
From 10.5.151.36 icmp_seq=21 Destination Host Unreachable
From 10.5.151.36 icmp_seq=22 Destination Host Unreachable
64 bytes from 10.5.150.244: icmp_seq=27 ttl=62 time=1.10 ms
64 bytes from 10.5.150.244: icmp_seq=28 ttl=62 time=0.750 ms
64 bytes from 10.5.150.244: icmp_seq=29 ttl=62 time=0.840 ms

tags: added: verification-train-done
removed: verification-train-needed
Dan Streetman (ddstreet)
description: updated
Revision history for this message
Robie Basak (racb) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello ITD27M01, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

The test case is verified on bionic-proposed and the external access to VM is restored once the router is re-enabled.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

The test case is verified on xenial-queens with queens-proposed packages and the external access to VM is restored once the router is re-enabled.

tags: added: verification-done verification-queens-done
removed: verification-needed verification-queens-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:15.3.3-0ubuntu1~cloud0
---------------

 neutron (2:15.3.3-0ubuntu1~cloud0) bionic-train; urgency=medium
 .
   * New stable point release for OpenStack Train (LP: #1923038).
   * d/p/revert-dvr-remove-control-plane-arp-updates.patch, d/p/0001-Fix-
     deletion-of-rfp-interfaces-when-router-is-re-ena.patch: Removed
     after fix landed upstream.
 .
 neutron (2:15.3.2-0ubuntu1~cloud2) bionic-train; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7

---------------
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium

  * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
    - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
      (LP: #1849098).

neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium

  * Do not initialize snat-ns twice (LP: #1850779)
    - d/p/0001-Do-not-initialize-snat-ns-twice.patch

neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium

  * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
    - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

 -- Seyeong Kim <email address hidden> Mon, 03 May 2021 17:15:28 +0900

Changed in neutron (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7~cloud0
---------------

 neutron (2:12.1.1-0ubuntu7~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium
 .
   * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
     - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
       (LP: #1849098).
 .
 neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium
 .
   * Do not initialize snat-ns twice (LP: #1850779)
     - d/p/0001-Do-not-initialize-snat-ns-twice.patch
 .
 neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium
 .
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

Changed in cloud-archive:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers