DVR: Inter Tenant Traffic between two networks and connected through a shared network not reachable with DVR routers

Bug #1751396 reported by Swaminathan Vasudevan on 2018-02-24
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Status tracked in Rocky
Pike
Undecided
Unassigned
Queens
Critical
Unassigned
Rocky
Critical
Unassigned
neutron
Undecided
Swaminathan Vasudevan
neutron (Ubuntu)
Status tracked in Cosmic
Artful
Undecided
Unassigned
Bionic
Critical
Unassigned
Cosmic
Critical
Unassigned

Bug Description

Inter Tenant Traffic between Two Tenants on two different private networks connected through a common shared network (created by Admin) is not route able through DVR routers

Steps to reproduce it:

(NOTE: No external, just shared network)
This is only reproducable in Multinode scenario. ( 1 Controller - 2 compute ).
Make sure that the two VMs are isolated in two different computes.

openstack network create --share shared_net

openstack subnet create shared_net_sn --network shared_net --subnet-range 172.168.10.0/24

openstack network create net_A
openstack subnet create net_A_sn --network net_A --subnet-range 10.1.0.0/24

openstack network create net_B
openstack subnet create net_B_sn --network net_B --subnet-range 10.2.0.0/24

openstack router create router_A

openstack port create --network=shared_net --fixed-ip subnet=shared_net_sn,ip-address=172.168.10.20 port_router_A_shared_net
openstack router add port router_A port_router_A_shared_net
openstack router add subnet router_A net_A_sn

openstack router create router_B
openstack port create --network=shared_net --fixed-ip subnet=shared_net_sn,ip-address=172.168.10.30 port_router_B_shared_net
openstack router add port router_B port_router_B_shared_net
openstack router add subnet router_B net_B_sn

openstack server create server_A --flavor m1.tiny --image cirros --nic net-id=net_A
openstack server create server_B --flavor m1.tiny --image cirros --nic net-id=net_B

Add static routes to the router.
openstack router set router_A --route destination=10.1.0.0/24,gateway=172.168.10.20
openstack router set router_B --route destination=10.2.0.0/24,gateway=172.168.10.30
```

Ping from one instance to the other times out

Ubuntu SRU details:
-------------------
[Impact]
See above

[Test Case]
Deploy OpenStack with dvr enabled and then follow the steps above.

[Regression Potential]
The patches that are backported have already landed upstream in the corresponding stable branches, helping to minimize any regression potential.

Changed in neutron:
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/547696

Changed in neutron:
assignee: nobody → Swaminathan Vasudevan (swaminathan-vasudevan)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/547696
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Submitter: Zuul
Branch: master

commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/554644
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=02d31ffb8abb157dff02fc977241ebdbf8ead89c
Submitter: Zuul
Branch: stable/queens

commit 02d31ffb8abb157dff02fc977241ebdbf8ead89c
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

tags: added: in-stable-queens
Changed in neutron (Ubuntu Artful):
status: New → Triaged
Changed in neutron (Ubuntu Bionic):
status: New → Triaged
Changed in neutron (Ubuntu Artful):
importance: Undecided → High
Changed in neutron (Ubuntu Bionic):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:12.0.1-0ubuntu1

---------------
neutron (2:12.0.1-0ubuntu1) bionic; urgency=medium

  * d/p/dvr-inter-tenant-traffic.patch: Cherry-picked from upstream
    stable/queens branch (LP: #1751396).
  * New stable point release for OpenStack Queens (LP: #1765138).

 -- Corey Bryant <email address hidden> Wed, 18 Apr 2018 12:07:48 -0400

Changed in neutron (Ubuntu Bionic):
status: Triaged → Fix Released

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-queens-needed

This issue was fixed in the openstack/neutron 12.0.2 release.

tags: added: neutron-proactive-backport-potential
Corey Bryant (corey.bryant) wrote :

12.0.2 is available in the Queens cloud archive.

Reviewed: https://review.openstack.org/558585
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6956821764e81c61f4c3620a2a6753394b5bc69a
Submitter: Zuul
Branch: stable/pike

commit 6956821764e81c61f4c3620a2a6753394b5bc69a
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

Reviewed: https://review.openstack.org/578112
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f65e7ba05d0502bdc7163a6221ed8138ac183055
Submitter: Zuul
Branch: stable/ocata

commit f65e7ba05d0502bdc7163a6221ed8138ac183055
Author: Swaminathan Vasudevan <email address hidden>
Date: Fri Feb 23 16:22:33 2018 -0800

    DVR: Inter Tenant Traffic between networks not possible with shared net

    Inter Tenant Traffic between two different networks that belong
    to two different Tenants is not possible when connected through
    a shared network that are internally connected through DVR
    routers.

    This issue can be seen in multinode environment where there
    is network isolation.

    The issue is, we have two different IP for the ports that are
    connecting the two routers and DVR does not expose the router
    interfaces outside a compute and is blocked by ovs tunnel bridge
    rules.

    This patch fixes the issue by not applying the DVR specific
    rules in the tunnel-bridge to the shared network ports that
    are connecting the routers.

    Closes-Bug: #1751396
    Change-Id: I0717f29209f1354605d2f4128949ddbaefd99629
    (cherry picked from commit d019790fe436b72cb05b8d0ff1f3a62ebd9e9bee)

tags: added: in-stable-ocata
tags: removed: neutron-proactive-backport-potential
Arjun Baindur (abaindur) wrote :

Hi, this appears to have broken DVR functionality for a simple shared tenant network. The DVR flows translating the DVR macs are missing on the physical bridges now for a VLAN based network. This tenant network is shared, and only attached to 1 router.

As you can see port_shared_only is set to True here because its shared and not an external network.

2018-07-25 15:53:02.386 673 DEBUG neutron.api.rpc.handlers.dvr_rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] neutron.api.rpc.handlers.dvr_rpc.DVRServerRpcApi method get_network_info_for_id called with arguments (<neutron_lib.context.ContextBase object at 0x7f67cb87fd50>, u'3f6ec232-7649-4639-b828-c3af9960481b') {} wrapper /opt/pf9/pf9-neutron/lib/python2.7/site-packages/oslo_log/helpers.py:66
2018-07-25 15:53:02.779 673 INFO neutron.common.rpc [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] PF9_EVENT: RPC_get_network_info_for_id elapsed: 0.39 seconds
2018-07-25 15:53:02.781 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: port_net_info = [{u'provider:physical_network': u'dogfood-internal', u'ipv6_address_scope': None, u'dns_domain': u'', u'revision_number': 7, u'port_security_enabled': True, u'mtu': 1500, u'id': u'3f6ec232-7649-4639-b828-c3af9960481b', u'router:external': False, u'availability_zone_hints': [], u'availability_zones': [u'nova'], u'ipv4_address_scope': None, u'shared': True, u'project_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'status': u'ACTIVE', u'subnets': [u'3707b250-b6f5-4701-9b17-01a8f288c17a'], u'description': None, u'tags': [], u'provider:segmentation_id': 795, u'name': u'df-tenant-795-dont-delete', u'admin_state_up': True, u'tenant_id': u'f175f441ebbb4c2b8fedf6469d6415fc', u'provider:network_type': u'vlan', u'vlan_transparent': None}]
2018-07-25 15:53:02.782 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: net_shared_only = True
2018-07-25 15:53:02.784 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: Not applying DVR rules to tunnel bridge because 3f6ec232-7649-4639-b828-c3af9960481b is a shared network
2018-07-25 15:53:02.785 673 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_dvr_neutron_agent [req-7ebadd38-bce9-4d2c-971a-50894cc78046 - - - - -] ARJUN3: vlan = 1, port.vif_mac = fa:16:3e:42:a2:ec, dvr_mac = fa:16:3f:1a:bf:de

I think we need more checks here - seems like it was intended only for a shared network that acts as a link between 2 routers, but doesnt attach any compute ports?

Arjun Baindur (abaindur) wrote :

Logged https://bugs.launchpad.net/neutron/+bug/1783654 to address the missing DVR flows on a shared VLAN tenant network

Reviewed: https://review.openstack.org/595496
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Submitter: Zuul
Branch: master

commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f

Reviewed: https://review.openstack.org/596402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Submitter: Zuul
Branch: stable/queens

commit 2d47cb7d9a8ab021ccbc08713b32ccddf6e1baa7
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

tags: added: neutron-proactive-backport-potential
Corey Bryant (corey.bryant) wrote :

Moving status back to New for Ubuntu due to reverted patch.

Changed in neutron (Ubuntu):
status: Fix Released → New
status: New → Triaged
Corey Bryant (corey.bryant) wrote :

s/New/Triaged

Changed in neutron (Ubuntu Bionic):
status: Fix Released → Triaged
Changed in cloud-archive:
status: Fix Committed → Triaged
Corey Bryant (corey.bryant) wrote :

Marking Artful as invalid since it is EOL.

Changed in neutron (Ubuntu Artful):
status: Triaged → Invalid
Changed in neutron (Ubuntu Artful):
importance: High → Critical
Changed in neutron (Ubuntu Cosmic):
importance: High → Critical
Changed in neutron (Ubuntu Artful):
importance: Critical → Undecided
Changed in neutron (Ubuntu Bionic):
importance: High → Critical
Corey Bryant (corey.bryant) wrote :

This does not affect Ubuntu Pike because neutron 11.0.5 does not include the original patches that are being reverted.

description: updated
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1

---------------
neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium

  * New stable point release for OpenStack Rocky.
  * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
    revert DVR regressions (LP: #1751396)
  * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
    DVR regression (LP: #1783654).

 -- Corey Bryant <email address hidden> Tue, 02 Oct 2018 17:18:19 -0400

Changed in neutron (Ubuntu Cosmic):
status: Triaged → Fix Released
Changed in cloud-archive:
status: Triaged → Fix Committed

Hello Swaminathan, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.0.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in neutron (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Corey Bryant (corey.bryant) wrote :

Hello Swaminathan, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:13.0.1-0ubuntu1~cloud0
---------------

 neutron (2:13.0.1-0ubuntu1~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 neutron (2:13.0.1-0ubuntu1) cosmic; urgency=medium
 .
   * New stable point release for OpenStack Rocky.
   * d/p/revert-dvr-add-error-handling.patch: Cherry-picked from upstream to
     revert DVR regressions (LP: #1751396)
   * d/p/revert-dvr-inter-tenant.patch: Cherry-picked from upstream to revert
     DVR regression (LP: #1783654).

Changed in cloud-archive:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/607345
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1a0d630d22180cbe602d56cfb308732add453de
Submitter: Zuul
Branch: stable/rocky

commit d1a0d630d22180cbe602d56cfb308732add453de
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Aug 23 06:10:42 2018 +0000

    Revert "DVR: Add error handling for get_network_info_for_id rpc call"

    This reverts commit c331b898e19c8125d005c09d8a6e247805e506a8.
    Related-Bug: #1751396

    Change-Id: I2e348091b5bdd0f3ef056dd108342989ce57062f
    (cherry picked from commit 267eea50fd9e9c74faa96d529a7ec2fd8aa8d51f)

tags: added: in-stable-rocky
Corey Bryant (corey.bryant) wrote :

Regression testing successful for bionic-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1318.6413 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 494.8999 sec.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Corey Bryant (corey.bryant) wrote :

Regression testing successful for queens-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1000.6584 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 465.0920 sec.

tags: added: verification-queens-done
removed: verification-queens-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers