Removing router interface causes router to stop routing between all

[Diko Parvanov]

[Impact]
Stumbled upon an issue where removing a DVR HA router interface renders all other subnets connected to that router to stop routing. VMs can't reach the HA port (IP) of the router (ping).

Worked around this by:
openstack router set --disabled <ROUTER>
openstack router set --enable <ROUTER>

This has happened more than once in the current deployment
 - cloud:bionic-stein
 - neutron 2:14.0.4-0ubuntu1~cloud1

[Test Case]
1. Reproducing the issue

1a. Deploy openstack using stsstack-bundles
    https://launchpad.net/stsstack-bundles

1b. Run the test script lp1892405_reproducer from comment #10

    The script does the following (Detailed steps in comment #4)
    - Create 3 projects P1, P2, P3
    - Create a router and network in each project, say R1,R2,R3 and
      N1,N2,N3
    - Cross-connect networks by adding ports to router.
    - Launch VMs on N1, N2 (Ensure VMs are landed on 2 different compute
      nodes)
    - ping from VM1 -> VM2 should be successful
    - Detach leg from N1 -> N3
    - Check for any packet loss during ping from VM1 -> VM2

    The script output shows the ping output from VM1 -> VM2 and there will
    be packet loss

2. Install the package with fixed code

3. Confirm bug have been fixed

3a. Cleanup of projects P1,P2,P3 and associated resources created in 1b
    Re-enable the hypervisor which is disabled as part of 1b script.
    Commands for the cleanup:
    openstack server list --all-projects -c ID -f value | xargs openstack server delete
    openstack router remove port P2-router to-n2
    openstack router remove port P1-router from-n2
    openstack router remove port P1-router from-n3
    for i in P1 P2 P3; do openstack subnet list --project $i -c ID -f value | xargs openstack router remove subnet $i-router; done
    for i in P1 P2 P3; do openstack router delete $i-router; done
    for i in P1 P2 P3; do openstack network list --project $i -c ID -f value | xargs openstack network delete; done
    openstack floating ip list -c ID -f value | xargs openstack floating ip delete
    for i in P1 P2 P3; do openstack project delete $i; done
    openstack compute service list --service nova-compute | grep disabled | awk '{print $6}' | xargs -I {} openstack compute service set --enable {} nova-compute

3b. Re-run the script 1b

    The script output shows the ping output from VM1 -> VM2 and there
    should not be any packet loss

[Where problems could occur]

Upstream CI ran all the functional and tempest test cases that involves deletion of DVR port connected to router which should cover the scenarios involving the code change.
Installation of new package will result in restart of neutron-openvswitch service and will take a few milliseconds to repopulate all the OVS flows.

[Peter Sabaini (peter-sabaini)]

Subscribing field-high, as core functionality is being impacted

[Edward Hope-Morley (hopem)]

I'm curious whether this a knock-on/related effect of bug 1892361

[Edward Hope-Morley (hopem)]

Sorry I meant bug 1891673

[Hemanth Nakkina (hemanth-n)]

The scenario seems to be caused when a network is connected to multiple DVR routers and when the non-gateway port is detached from the router. dvr-src mac flows are removed for all the VMs in the subnet on all the compute nodes [1].

Steps to reproduce:

1. Create 3 networks (n1, n2, n3) each connected to their internal routers (r1, r2, r3)

openstack network create n1
openstack subnet create --network n1 --dhcp --gateway 192.168.71.1 --subnet-range 192.168.71.0/24 --allocation-pool start=192.168.71.2,end=192.168.71.254 sn1
openstack router create --distributed --ha r1
openstack router add subnet r1 sn1

openstack network create n2
openstack subnet create --network n2 --dhcp --gateway 192.168.72.1 --subnet-range 192.168.72.0/24 --allocation-pool start=192.168.72.2,end=192.168.72.254 sn2
openstack router create --distributed --ha r2
openstack router add subnet r2 sn2

openstack network create n3
openstack subnet create --network n3 --dhcp --gateway 192.168.73.1 --subnet-range 192.168.73.0/24 --allocation-pool start=192.168.73.2,end=192.168.73.254 sn3
openstack router create --distributed --ha r3
openstack router add subnet r3 sn3

2. Connect n1, n2 by adding legs on r2, r1.
   Connect n1, n3 by adding legs on r3, r1.

openstack port create --network n1 to-n2
openstack router add port r2 to-n2
openstack port create --network n2 from-n2
openstack router add port r1 from-n2

openstack port create --network n1 to-n3
openstack router add port r3 to-n3
openstack port create --network n3 from-n3
openstack router add port r1 from-n3

3. Launch 2 VMs each on n1 and n2, lets say vm-n1-01, vm-n2-01
   Update security groups to allow icmp traffic.
   Ping from vm-n1-01 to vm-n2-01, and should be successful.

4. Detach the leg from n1 to n3 by removing n1 port on r3.

openstack router r3 remove port to-n3

5. Ping from vm-n1-01 to vm-n2-01 fails

   The ICMP echo request reaches vm-n2-01 and reply sent back. The ICMP echo reply receives on the source compute node and gets dropped at br-int since there is no dvr-src mac flow.

   Expectation: Removing a port on r3 should not affect traffic between n1 - n2.

The dvr-src mac rules should not be dropped when DVR router port is deleted unless it is the gateway port.

[1] https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py#L656-L664

[Hemanth Nakkina (hemanth-n)]

Mistake in the command mentioned in step 4 in #4 comment

Command should be:
openstack router remove port r3 to-n3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/752248

[Edward Hope-Morley (hopem)]

tested with proposed patch and i no longer see the problem.

[Hemanth Nakkina (hemanth-n)]

Reproducer script is modified in #10 to remove n1 leg from router3 (instead of n3 leg from router1).

[Edward Hope-Morley (hopem)]

thanks hemanth, confirmed i can repro with that modification - https://pastebin.ubuntu.com/p/38V68TkPsv/ - my vm mac is fa:16:3e:4a:46:c0 and can see that after removing n1 leg from r3 i hit the issue if not being able to reach vm on n2 from n1.

[Corey Bryant (corey.bryant)]

This patch has landed in upstream master and backports have been proposed for stable/ussuri and stable/victoria upstream.

[Corey Bryant (corey.bryant)]

This is fixed in hirsute-proposed now with neutron 2:17.0.0+git2020120911.b5ea767bc3-0ubuntu2.

[Corey Bryant (corey.bryant)]

https://launchpad.net/ubuntu/+source/neutron/2:17.0.0+git2020120911.b5ea767bc3-0ubuntu2

[Corey Bryant (corey.bryant)]

New versions of neutron with this fix have been uploaded to the groovy and focal unapproved queues:

https://launchpad.net/ubuntu/groovy/+queue?queue_state=1&queue_text=neutron
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=neutron

Hemanth, thanks for all your work on this!

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:17.0.0+git2020120911.b5ea767bc3-0ubuntu2

---------------
neutron (2:17.0.0+git2020120911.b5ea767bc3-0ubuntu2) hirsute; urgency=medium

  * d/p/fix-removal-of-dvr-src-mac-flows.patch: Cherry-picked from upstream to
    fix removal of dvr-src mac flows when non-gateway port on router is deleted
    (LP: #1892405).
  * d/control: Set min BD version of python3-eventlet to 0.30.0.
  * d/control: Set min BD version of python-oslo.db to 8.5.0+really.8.4.0 to
    align with upper-constraints.
  * d/rules: Add PYTHONWARNINGS to unit test execution inline with upstream.

 -- Corey Bryant <email address hidden> Mon, 04 Jan 2021 14:04:22 -0500

[Brian Murray (brian-murray)]

Hello Diko, or anyone else affected,

Accepted neutron into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:17.0.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Diko, or anyone else affected,

Accepted neutron into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:16.2.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Hemanth Nakkina (hemanth-n)]

Verified on groovy and focal. The test case is successful.
Attached the verification logs.

[Hemanth Nakkina (hemanth-n)]

Uploaded debdiffs for UCA train and UCA stein

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted neutron into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted neutron into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hi Hemanth, Can you propose the stein patch upstream? I think stable/stein is in extended maintenance now. I'd just like to get an upstream review of the conflicts if at all possible.

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Akihiro Motoki (amotoki)]

The fix in the neutron repo was merged last month. OpenStack gerrit no longer updates the launchpad status so we need to update the status manually.

[Hemanth Nakkina (hemanth-n)]

Verified both victoria-proposed, train-proposed and test case is working as expected.

Corey,
I will work on getting stein in upstream.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:16.2.0-0ubuntu2

---------------
neutron (2:16.2.0-0ubuntu2) focal; urgency=medium

  * d/p/fix-removal-of-dvr-src-mac-flows.patch: Cherry-picked from upstream to
    fix removal of dvr-src mac flows when non-gateway port on router is deleted
    (LP: #1892405).

 -- Corey Bryant <email address hidden> Thu, 07 Jan 2021 15:10:11 -0500

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:17.0.0-0ubuntu2

---------------
neutron (2:17.0.0-0ubuntu2) groovy; urgency=medium

  * d/gbp.conf: Create stable/victoria branch.
  * d/p/fix-removal-of-dvr-src-mac-flows.patch: Cherry-picked from upstream to
    fix removal of dvr-src mac flows when non-gateway port on router is deleted
    (LP: #1892405).

 -- Corey Bryant <email address hidden> Thu, 07 Jan 2021 15:10:07 -0500

[Corey Bryant (corey.bryant)]

@Hemanth, Thank you. By any chance can you test ussuri as well?

[Hemanth Nakkina (hemanth-n)]

Verified on bionic ussuri with ussuri-proposed package and the test case is successful.
Fix for stein is merged in upstream and the uploaded debdiff for UCA stein should be good for SRU.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:17.0.0-0ubuntu2~cloud0
---------------

 neutron (2:17.0.0-0ubuntu2~cloud0) focal-victoria; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:17.0.0-0ubuntu2) groovy; urgency=medium
 .
   * d/gbp.conf: Create stable/victoria branch.
   * d/p/fix-removal-of-dvr-src-mac-flows.patch: Cherry-picked from upstream to
     fix removal of dvr-src mac flows when non-gateway port on router is deleted
     (LP: #1892405).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:16.2.0-0ubuntu2~cloud0
---------------

 neutron (2:16.2.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:16.2.0-0ubuntu2) focal; urgency=medium
 .
   * d/p/fix-removal-of-dvr-src-mac-flows.patch: Cherry-picked from upstream to
     fix removal of dvr-src mac flows when non-gateway port on router is deleted
     (LP: #1892405).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:15.3.0-0ubuntu1~cloud3
---------------

 neutron (2:15.3.0-0ubuntu1~cloud3) bionic-train; urgency=medium
 .
   * d/p/fix-dvr-source-mac-flows.patch: Fix DVR source mac flows when non-gateway
     port on router is deleted (LP: #1892405).

[Corey Bryant (corey.bryant)]

Hemanth, thanks again for your work on this. I've uploaded a new version for stein based on your debdiff, however I updated the patch based on what landed upstream in stable/stein.

[Corey Bryant (corey.bryant)]

neutron 2:14.4.2-0ubuntu1~cloud1 is in stein-staging and waiting for 2:14.4.2-0ubuntu1~cloud0 to be verified and promoted to updates.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.1.0 release.

[Chris MacNaughton (chris.macnaughton)]

Verified on bionic stein with stein-proposed package and the test case is successful.

Additionally, tempest smoke was successful:

======
Totals
======
Ran: 97 tests in 680.7994 sec.
 - Passed: 89
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 953.6700 sec.

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Hemanth Nakkina (hemanth-n)]

Verified on bionic-stein with neutron package 2:14.4.2-0ubuntu1~cloud1 (stein-proposed)and the issue is fixed.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:14.4.2-0ubuntu1~cloud1
---------------

 neutron (2:14.4.2-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   * d/p/fix-dvr-source-mac-flows.patch: Fix DVR source mac flows when non-gateway
     port on router is deleted (LP: #1892405).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron stein-eol release.

[Nate Bill (nbill)]

Possible regression - I am seeing this on 2023.1 with a HA router. When detaching a non-gateway port the entire router stops routing traffic.