DVR: Inter Tenant Traffic between two networks and connected through a shared network not reachable with DVR routers

[Swaminathan Vasudevan]

Inter Tenant Traffic between Two Tenants on two different private networks connected through a common shared network (created by Admin) is not route able through DVR routers

Steps to reproduce it:

(NOTE: No external, just shared network)
This is only reproducable in Multinode scenario. ( 1 Controller - 2 compute ).
Make sure that the two VMs are isolated in two different computes.

openstack network create --share shared_net

openstack subnet create shared_net_sn --network shared_net --subnet-range 172.168.10.0/24

openstack network create net_A
openstack subnet create net_A_sn --network net_A --subnet-range 10.1.0.0/24

openstack network create net_B
openstack subnet create net_B_sn --network net_B --subnet-range 10.2.0.0/24

openstack router create router_A

openstack port create --network=shared_net --fixed-ip subnet=shared_net_sn,ip-address=172.168.10.20 port_router_A_shared_net
openstack router add port router_A port_router_A_shared_net
openstack router add subnet router_A net_A_sn

openstack router create router_B
openstack port create --network=shared_net --fixed-ip subnet=shared_net_sn,ip-address=172.168.10.30 port_router_B_shared_net
openstack router add port router_B port_router_B_shared_net
openstack router add subnet router_B net_B_sn

openstack server create server_A --flavor m1.tiny --image cirros --nic net-id=net_A
openstack server create server_B --flavor m1.tiny --image cirros --nic net-id=net_B

Add static routes to the router.
openstack router set router_A --route destination=10.1.0.0/24,gateway=172.168.10.20
openstack router set router_B --route destination=10.2.0.0/24,gateway=172.168.10.30
```

Ping from one instance to the other times out

Ubuntu SRU details:
-------------------
[Impact]
See above

[Test Case]
Deploy OpenStack with dvr enabled and then follow the steps above.

[Regression Potential]
The patches that are backported have already landed upstream in the corresponding stable branches, helping to minimize any regression potential.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/547696

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547696
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Submitter: Zuul
Branch: master

commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/554644

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554644
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=02d31ffb8abb157dff02fc977241ebdbf8ead89c
Submitter: Zuul
Branch: stable/queens

commit 02d31ffb8abb157dff02fc977241ebdbf8ead89c
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/558585

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:12.0.1-0ubuntu1

---------------
neutron (2:12.0.1-0ubuntu1) bionic; urgency=medium

  * d/p/dvr-inter-tenant-traffic.patch: Cherry-picked from upstream
    stable/queens branch (LP: #1751396).
  * New stable point release for OpenStack Queens (LP: #1765138).

 -- Corey Bryant <email address hidden> Wed, 18 Apr 2018 12:07:48 -0400

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

[Corey Bryant (corey.bryant)]

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.2 release.

[Corey Bryant (corey.bryant)]

12.0.2 is available in the Queens cloud archive.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/578112

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558585
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6956821764e81c61f4c3620a2a6753394b5bc69a
Submitter: Zuul
Branch: stable/pike

commit 6956821764e81c61f4c3620a2a6753394b5bc69a
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/578112
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f65e7ba05d0502bdc7163a6221ed8138ac183055
Submitter: Zuul
Branch: stable/ocata

commit f65e7ba05d0502bdc7163a6221ed8138ac183055
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

[Arjun Baindur (abaindur)]

Hi, this appears to have broken DVR functionality for a simple shared tenant network. The DVR flows translating the DVR macs are missing on the physical bridges now for a VLAN based network. This tenant network is shared, and only attached to 1 router.

As you can see port_shared_only is set to True here because its shared and not an external network.

2018-07-25 15:53:02.386 673 DEBUG neutron.api.rpc.handlers.dvr_rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] neutron.api.rpc.handlers.dvr_rpc.DVRServerRpcApi method get_network_info_for_id called with arguments (<neutron_lib.context.ContextBase object at 0x7f67cb87fd50>, u'3f6ec232-7649-4639-b828-c3af9960481b') {} wrapper /opt/pf9/pf9-neutron/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-07-25 15:53:02.779 673 INFO neutron.common.rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] PF9_EVENT: RPC_get_network_info_for_id elapsed: 0.39 seconds
2018-07-25 15:53:02.781 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: port_net_info = [{u'provider:physical_network': u'dogfood-internal', u'ipv6_address_scope': None, u'dns_domain': u'', u'revision_number': 7, u'port_security_enabled': True, u'mtu': 1500, u'id': u'3f6ec232-7649-4639-b828-c3af9960481b', u'router:external': False, u'availability_zone_hints': [], u'availability_zones': [u'nova'], u'ipv4_address_scope': None, u'shared': True, u'project_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'status': u'ACTIVE', u'subnets': [u'3707b250-b6f5-4701-9b17-01a8f288c17a'], u'description': None, u'tags': [], u'provider:segmentation_id': 795, u'name': u'df-tenant-795-dont-delete', u'admin_state_up': True, u'tenant_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'provider:network_type': u'vlan', u'vlan_transparent': None}]
2018-07-25 15:53:02.782 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: net_shared_only = True
2018-07-25 15:53:02.784 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: Not applying DVR rules to tunnel bridge because 3f6ec232-7649-4639-b828-c3af9960481b is a shared network
2018-07-25 15:53:02.785 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: vlan = 1, port.vif_mac = fa:16:3e:42:a2:ec, dvr_mac = fa:16:3f:1a:bf:de

I think we need more checks here - seems like it was intended only for a shared network that acts as a link between 2 routers, but doesnt attach any compute ports?

[Arjun Baindur (abaindur)]

Logged https://bugs.launchpad.net/neutron/+bug/1783654 to address the missing DVR flows on a shared VLAN tenant network

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/595496

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/595496
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Submitter: Zuul
Branch: master

commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/596402

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Submitter: Zuul
Branch: stable/queens

commit 2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

[Corey Bryant (corey.bryant)]

Moving status back to New for Ubuntu due to reverted patch.

[Corey Bryant (corey.bryant)]

s/New/Triaged

[Corey Bryant (corey.bryant)]

Marking Artful as invalid since it is EOL.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607345

[Corey Bryant (corey.bryant)]

This does not affect Ubuntu Pike because neutron 11.0.5 does not include the original patches that are being reverted.

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1

---------------
neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium

  * New stable point release for OpenStack Rocky.
  * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
    revert DVR regressions (LP: #1751396)
  * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
    DVR regression (LP: #1783654).

 -- Corey Bryant <email address hidden> Tue, 02 Oct 2018 17:18:19 -0400

[Timo Aaltonen (tjaalton)]

Hello Swaminathan, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.0.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1~cloud0
---------------

 neutron (2:13.0.1-0ubuntu1~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium
 .
   * New stable point release for OpenStack Rocky.
   * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
     revert DVR regressions (LP: #1751396)
   * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
     DVR regression (LP: #1783654).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607345
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1a0d630d22180cbe602d56cfb308732add453de
Submitter: Zuul
Branch: stable/rocky

commit d1a0d630d22180cbe602d56cfb308732add453de
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

[Corey Bryant (corey.bryant)]

Regression testing successful for bionic-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1318.6413 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 494.8999 sec.

[Corey Bryant (corey.bryant)]

Regression testing successful for queens-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1000.6584 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 465.0920 sec.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/611396

[Brian Murray (brian-murray)]

I don't think the regression testing is sufficient for this SRU verification, please follow the test case in the description to verify that this is fixed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/611396
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=eb7bb07184bfcc2f64cbf53f9d69891cd7261eea
Submitter: Zuul
Branch: stable/pike

commit eb7bb07184bfcc2f64cbf53f9d69891cd7261eea
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)
    (cherry picked from commit 2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.

[Corey Bryant (corey.bryant)]

This SRU is no longer needed as we're uploading neutron 12.0.5 to bionic (queens) which includes this fix.

[Brian Murray (brian-murray)]

Thank you for taking the time to verify this stable release fix. We have noticed that you have used the verification-done tag for marking the bug as verified and would like to point out that due to a recent change in SRU bug verification policy fixes now have to be marked with per-release tags (i.e. verification-done-$RELEASE). Please remove the verification-done tag and add one for the release you have tested the package in. Thank you!

https://wiki.ubuntu.com/StableReleaseUpdates#Verification

[Corey Bryant (corey.bryant)]

This bug has been fixed in the neutron 12.0.5 stable point release via https://bugs.launchpad.net/cloud-archive/+bug/1795424. neutron 12.0.5 was tested with a tempest smoke test run against a juju-deployed openstack as described in https://wiki.ubuntu.com/OpenStackUpdates.
Please see test results at: https://bugs.launchpad.net/cloud-archive/+bug/1795424/comments/17

Thanks,
Corey

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.