DVR:Removing an interface from router with multiple interfaces and with ext gw set does not remove interface from SNAT namespace

Bug #1479130 reported by Hardik Italia
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Swaminathan Vasudevan

Bug Description

Steps to reproduce:
1) Create one private and one public network.
2) Create DVR Router.
3) Add internal interface to router.
4) Set gateway to router. (qrouter & snat namespace should be created).
5) Remove internal interface from router (by port or by subnet)
6) Notice that corresponding SNAT interface for the internal network from SNAT namespace is still there.

So if we add internal interface again to a router then 2 SNAT interfaces for internal network will be there in the SNAT Namespace, which breaks external traffic for private subnet.

$ neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| 6a180ace-23a5-4300-89b2-e54872b4994c | n1 | f16081e0-5674-4caf-aeef-19f1ca3ab4cf 192.168.20.0/24 |
| acf1512c-683b-435c-a161-5c5eba916fa0 | ext-net | 8bf3aa4a-8791-44d1-8a7a-0c99a9412c09 10.10.20.0/24 |
+--------------------------------------+---------+------------------------------------------------------+

$ neutron router-list
+--------------------------------------+------+-----------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+------+-----------------------+-------------+-------+
| 4948fdfa-6f67-4ede-8e9a-dc960c08b4fd | r1 | null | True | False |
+--------------------------------------+------+-----------------------+-------------+-------+

$ neutron router-interface-add r1 s1
Added interface 59f3fd7b-5125-41a3-95fe-368890f955e4 to router r1.

$ neutron router-gateway-set r1 ext-net
Set gateway for router r1

$ ip netns
snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd
qrouter-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd

$ neutron router-interface-delete r1 s1
Removed interface from router r1

It remove interface from qrouter namespace

$ sudo ip netns exec qrouter-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Not removing sg interface from sname namespace.

 sudo ip netns exec snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

qg-9c6eb6ec-17 Link encap:Ethernet HWaddr fa:16:3e:77:4c:43
          inet addr:10.10.20.107 Bcast:10.10.20.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe77:4c43/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:1300 (1.3 KB)

sg-4f5377ff-fc Link encap:Ethernet HWaddr fa:16:3e:ae:ac:d2
          inet addr:192.168.20.3 Bcast:192.168.20.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:feae:acd2/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:992 (992.0 B) TX bytes:952 (952.0 B)

Re-adding internal interface to router will have 2 sg ports inside the SNAT namespace.

$ neutron router-interface-add r1 s1
Added interface 57d66312-c222-4df2-9120-273a9a540925 to router r1.

$ sudo ip netns exec snat-4948fdfa-6f67-4ede-8e9a-dc960c08b4fd ifconfig
lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

qg-9c6eb6ec-17 Link encap:Ethernet HWaddr fa:16:3e:77:4c:43
          inet addr:10.10.20.107 Bcast:10.10.20.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe77:4c43/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:1300 (1.3 KB)

sg-4f5377ff-fc Link encap:Ethernet HWaddr fa:16:3e:ae:ac:d2
          inet addr:192.168.20.3 Bcast:192.168.20.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:feae:acd2/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:992 (992.0 B) TX bytes:952 (952.0 B)

sg-9ea241ad-af Link encap:Ethernet HWaddr fa:16:3e:8c:ac:bb
          inet addr:192.168.20.4 Bcast:192.168.20.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe8c:acbb/64 Scope:Link
          UP BROADCAST RUNNING MTU:1500 Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:174 (174.0 B) TX bytes:964 (964.0 B)

Note: This issue is noticed in Kilo and later.

Changed in neutron:
assignee: nobody → Ashok Chippa (a-n-chippa)
Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Verified the issue is confirmed.
This issue has been introduced in the l3 agent refactor.
I have a fix and will upload a patch.

Changed in neutron:
status: New → Confirmed
Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

The internal CSNAT ports are not cached in the agent anymore and so the "get_snat_ports_for_the_internal_port" function fails, since it could not retrieve any CSNAT ports for the given subnet.

summary: - DVR:Removing interface from router with ext gw set does not remove
- interface from SNAT namespace
+ DVR:Removing an interface from router with multiple interfaces and with
+ ext gw set does not remove interface from SNAT namespace
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/215329

Changed in neutron:
assignee: Ashok Chippa (a-n-chippa) → Swaminathan Vasudevan (swaminathan-vasudevan)
status: Confirmed → In Progress
Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/216890

Changed in neutron:
assignee: Swaminathan Vasudevan (swaminathan-vasudevan) → Carl Baldwin (carl-baldwin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/216902

Changed in neutron:
assignee: Carl Baldwin (carl-baldwin) → Swaminathan Vasudevan (swaminathan-vasudevan)
Changed in neutron:
milestone: none → liberty-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/216890
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3a9e778399af8380b11c968da39e08b4a97a9f1f
Submitter: Jenkins
Branch: master

commit 3a9e778399af8380b11c968da39e08b4a97a9f1f
Author: Carl Baldwin <email address hidden>
Date: Tue Aug 25 22:32:50 2015 +0000

    Add snat ports cache to dvr router

    This reverses the effect of [1] but in a way that works with the
    current structure of the code and keeps DVR details in DVR classes

    [1] https://review.openstack.org/#/c/200293

    Change-Id: Ia8468881de6538882d4a14725b55db53e23d2e4c
    Closes-Bug: #1479130

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/216902
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=34a329b4de7c801c15b3c214cc2b122ac82d0b72
Submitter: Jenkins
Branch: master

commit 34a329b4de7c801c15b3c214cc2b122ac82d0b72
Author: Swaminathan Vasudevan <email address hidden>
Date: Tue Aug 25 16:24:05 2015 -0700

    Add a functional test to validate dvr snat namespace

    Add a functional test to validate the dvr snat
    namespace and its internal interfaces when internal
    networks are removed and added.

    Change-Id: Id44f5e5899e959be53b09e6f9bc732f553ae9a5a
    Related-Bug: #1479130

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Swaminathan Vasudevan (<email address hidden>) on branch: master
Review: https://review.openstack.org/215329
Reason: Another patch related to this fix got merged and so I will go ahead and abandon this patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (feature/pecan)

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/224357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (feature/pecan)
Download full text (73.6 KiB)

Reviewed: https://review.openstack.org/224357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdc3431ccd219accf6a795079d9b67b8656eed8e
Submitter: Jenkins
Branch: feature/pecan

commit fe236bdaadb949661a0bfb9b62ddbe432b4cf5f1
Author: Miguel Angel Ajo <email address hidden>
Date: Thu Sep 3 15:40:12 2015 +0200

    No network devices on network attached qos policies

    Network devices, like internal router legs, or dhcp ports
    should not be affected by bandwidth limiting rules.

    This patch disables application of network attached policies
    to network/neutron owned ports.

    Closes-bug: #1486039
    DocImpact

    Change-Id: I75d80227f1e6c4b3f5fa7762b8dc3b0c0f1abd46

commit db4a06f7caa20a4c7879b58b20e95b223ed8eeaf
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Sep 16 10:04:32 2015 +0000

    Use tempest-lib's token_client

    Now tempest-lib provides token_client modules as library and the
    interface is stable. So neutron repogitory doesn't need to contain
    these modules.
    This patch makes neutron use tempest-lib's token_client and removes
    the own modules for the maintenance.

    Change-Id: Ieff7eb003f6e8257d83368dbc80e332aa66a156c

commit 78aed58edbe6eb8a71339c7add491fe9de9a0546
Author: Jakub Libosvar <email address hidden>
Date: Thu Aug 13 09:08:20 2015 +0000

    Fix establishing UDP connection

    Previously, in establish_connection() for UDP protocol data were sent
    but never read on peer socket. That lead to successful read on peer side
    if this connection was filtered. Having constant testing string masked
    this issue as we can't distinguish to which test of connectivity data
    belong.

    This patch makes unique data string per test_connectivity() and
    also makes establish_connection() to create an ASSURED entry in
    conntrack table. Finally, in last test after firewall filter was
    removed, connection is re-established in order to avoid troubles with
    terminated processes or TCP continuing sending packets which weren't
    successfully delivered.

    Closes-Bug: 1478847
    Change-Id: I2920d587d8df8d96dc1c752c28f48ba495f3cf0f

commit e6292fcdd6262434a7b713ad8802db6bc8a6d3dc
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Sep 16 13:20:51 2015 +0900

    ovsdb: Fix a few docstring

    Change-Id: I53e1e21655b28fe5da60e58aeeb7cbbd103ae014

commit c22949a4449d96a67caa616290cf76b67b182917
Author: fumihiko kakuma <email address hidden>
Date: Wed Sep 16 11:52:59 2015 +0900

    Remove requirements.txt for the ofagent mechanism driver

    It is no longer used.

    Related-Blueprint: core-vendor-decomposition
    https://blueprints.launchpad.net/neutron/+spec/core-vendor-decomposition

    Change-Id: Ib31fb3febf8968e50d86dd66e1e6e1ea2313f8ac

commit d1d4de19d85f961d388c91e70f31b3bafec418c5
Author: Kevin Benton <email address hidden>
Date: Thu Sep 3 20:25:57 2015 -0700

    Always return iterables in L3 get_candidates

    The caller of this function expects iterables.

    Closes-Bug: #1494996
    Change-Id: I3d103e63f4e127a77268502415c0ddb0d804b54a

commit 1ad6ac448067306...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Doug Wiegley (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/224334

Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-rc1 → 7.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.