Removing an interface by port from a DVR router deletes all SNAT ports.

Bug #1443524 reported by Assaf Muller
28
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Oleg Bondarev

Bug Description

Steps to reproduce:
0) Multinode setup
1) Create distributed router
2) Add two internal interfaces
3) Set the router as gateway

At this point the router has 5 ports: Two internal, one external, and two owned by DVR SNAT

4) Now delete the first internal interface (By port, not by subnet).

Three ports were deleted instead of two: The internal port, its SNAT port, and the SNAT port of another interface.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Will check it out

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :

Assaf, in your message above for the "Second" bug that you have stated, you mentioned repeat steps from 0-3 and then in step 4. is same as the first one.
I assume this is a typo, can you update on what behavior you were seeing and how to reproduce it.

Revision history for this message
Assaf Muller (amuller) wrote :

@Swami, step 4 the first time is to remove the interface by subnet, and the second time by port.

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :
Download full text (7.8 KiB)

Assaf: Here is the test that I did for the "First issue" that you explained.

I don't see the "SNAT" namespace getting deleted. But I still see other interface ports available in the namespace.

stack@ubuntu:~/devstack$ neutron port-list
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 2dcac98d-8fc4-4645-9f7b-8f1b6e0b0ef0 | | fa:16:3e:75:2d:84 | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.1"} |
| 74a16561-3d48-4222-a686-3f6bbad83b0e | | fa:16:3e:c7:40:8e | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.3"} |
| 8f7cc47b-72cd-47cb-aa0b-d14f4e421cad | | fa:16:3e:61:7f:68 | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.2"} |
| a1cc4f97-662b-4fb5-a8ab-43b49bbf9faa | | fa:16:3e:f1:35:80 | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.1"} |
| a441ad7e-3dc1-4282-8be1-6b101cb2a811 | | fa:16:3e:db:97:1c | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.2"} |
| a4c68db6-ffab-44f2-80c2-0101b65f1c4c | | fa:16:3e:52:64:7f | {"subnet_id": "f0cda988-6981-4e1c-9382-e4950a6a9e25", "ip_address": "192.168.99.2"} |
| d5f31648-adf7-48a9-a6fd-df7c63c986cb | | fa:16:3e:66:2c:49 | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.3"} |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$ neutron subnet-list
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
| 4013195e-8384-414b-9c92-9ccbab1e05e8 | subnet1 | 10.1.0.0/24 | {"start": "10.1.0.2", "end": "10.1.0.254"} |
| ec7c96b2-45ad-4ae1-82de-1655c607a98e | private-subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| f0cda988-6981-4e1c-9382-e4950a6a9e25 | public-subnet | 192.168.99.0/24 | {"start": "192.168.99.2", "end": "192.168.99.254"} |
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
stack@ubuntu:~/devstack$ neutron subnet-delete private-subnet
Unable to complete operation on subnet ec7c96b2-45ad-4ae1-82de-1655c607a98e. One or more ports have an IP allocation from this subnet.
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$
stack@ubuntu:~/dev...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/173001

Changed in neutron:
assignee: nobody → Assaf Muller (amuller)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/173002

Revision history for this message
Assaf Muller (amuller) wrote : Re: Removing one internal interface from a DVR router deletes all SNAT ports

Swami, if you delete the interface by subnet, the SNAT portion of the router will be unscheduled if it's host is not hosting VM, DHCP or LB ports (For example if it's dvr_snat l3 agent without a DHCP, nova compute or LB agent on it). You probably had something else running on your dvr_snat node.

The second bug is that deleting a router interface by port will delete all SNAT ports of the router and not just the one corresponding to the interface that was deleted (This is trivial to reproduce).

Revision history for this message
Swaminathan Vasudevan (swaminathan-vasudevan) wrote :
Download full text (9.4 KiB)

The second bug that you have stated with respect to the "remove interface by port" deletes both the "csnat" ports. That is a valid bug.

Here are the console messages for it..

stack@ubuntu:~/devstack$ neutron router-interface-add router1 private-subnet
Added interface c8a8799c-6a2b-4cd9-913b-28b551da413f to router router1.
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$ neutron subnet-list
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
| 4013195e-8384-414b-9c92-9ccbab1e05e8 | subnet1 | 10.1.0.0/24 | {"start": "10.1.0.2", "end": "10.1.0.254"} |
| ec7c96b2-45ad-4ae1-82de-1655c607a98e | private-subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
| f0cda988-6981-4e1c-9382-e4950a6a9e25 | public-subnet | 192.168.99.0/24 | {"start": "192.168.99.2", "end": "192.168.99.254"} |
+--------------------------------------+----------------+-----------------+----------------------------------------------------+
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$
stack@ubuntu:~/devstack$ neutron port-list
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
| 59cb8ab6-05ba-4e65-ac9f-d6dcdb94ca62 | | fa:16:3e:c3:32:da | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.4"} |
| 74a16561-3d48-4222-a686-3f6bbad83b0e | | fa:16:3e:c7:40:8e | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.3"} |
| 8f7cc47b-72cd-47cb-aa0b-d14f4e421cad | | fa:16:3e:61:7f:68 | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.2"} |
| a1cc4f97-662b-4fb5-a8ab-43b49bbf9faa | | fa:16:3e:f1:35:80 | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.1"} |
| a441ad7e-3dc1-4282-8be1-6b101cb2a811 | | fa:16:3e:db:97:1c | {"subnet_id": "4013195e-8384-414b-9c92-9ccbab1e05e8", "ip_address": "10.1.0.2"} |
| a4c68db6-ffab-44f2-80c2-0101b65f1c4c | | fa:16:3e:52:64:7f | {"subnet_id": "f0cda988-6981-4e1c-9382-e4950a6a9e25", "ip_address": "192.168.99.2"} |
| c8a8799c-6a2b-4cd9-913b-28b551da413f | | fa:16:3e:7b:e3:d2 | {"subnet_id": "ec7c96b2-45ad-4ae1-82de-1655c607a98e", "ip_address": "10.0.0.1"} |
+--------------------------------------+------+-------------------+-------------------------------------------------------------------------------------+
stack@ubuntu:~/devstack$ sudo ip netns exec snat-1c44f751-c2ed-4dd9-b81e-9ebec8c7811...

Read more...

Assaf Muller (amuller)
summary: - Removing one internal interface from a DVR router deletes all SNAT ports
+ Removing an interface by port from a DVR router deletes all SNAT ports
description: updated
Assaf Muller (amuller)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/177992

Changed in neutron:
assignee: Assaf Muller (amuller) → David Pinheiro (davrodpin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/kilo)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/177992
Reason: Abandoning since there is no way the patch with the Change-Id goes in. It must be the same as for master patch.

Changed in neutron:
assignee: David Pinheiro (davrodpin) → Swaminathan Vasudevan (swaminathan-vasudevan)
Changed in neutron:
importance: Undecided → High
Changed in neutron:
assignee: Swaminathan Vasudevan (swaminathan-vasudevan) → Oleg Bondarev (obondarev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/173001
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0b5b6c7e746839d3f9bece6e782092bca3c9725e
Submitter: Jenkins
Branch: master

commit 0b5b6c7e746839d3f9bece6e782092bca3c9725e
Author: Assaf Muller <email address hidden>
Date: Mon Apr 13 13:26:06 2015 -0400

    Fix DVR interface delete by port when gateway is set

    When removing a DVR interface by port, the subnet_id
    passed to delete_csnat_router_interface_ports is None,
    and so it deletes all the DVR SNAT ports for the
    router.

    This patch fixes this issue by passing in the right
    subnet_id to the delete_csnat_router_interface_ports.

    Change-Id: I16735195c6575454876acd0e99ef45f382963566
    Closes-Bug: #1443524
    Co-Authored-By: Swaminathan Vasudevan <email address hidden>
    Co-Authored-By: Oleg Bondarev <email address hidden>

Changed in neutron:
status: In Progress → Fix Committed
summary: Removing an interface by port from a DVR router deletes all SNAT ports
+ even if the SNAT port does not belong to the ports subnet.
summary: - Removing an interface by port from a DVR router deletes all SNAT ports
- even if the SNAT port does not belong to the ports subnet.
+ Removing an interface by port from a DVR router deletes all SNAT ports.
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/218710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (155.6 KiB)

Reviewed: https://review.openstack.org/218710
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2c5f44e1b3bd4ed8a0b7232fd293b576cc8c1c87
Submitter: Jenkins
Branch: feature/pecan

commit f35d1c5c50dccbef1a2e079f967b82f0df0e22e9
Author: Adelina Tuvenie <email address hidden>
Date: Thu Aug 27 02:27:28 2015 -0700

    Fixes wrong neutron Hyper-V Agent name in constants

    Change Id03fb147e11541be309c1cd22ce27e70fadc28b5 moved the
    AGENT_TYPE_HYPERV constant from common.constants to
    plugins.ml2.drivers.hyperv.constants but change the value of the
    constant from 'HyperV agent' to 'hyperv'. This patch changes
    the name back to 'HyperV agent'

    Change-Id: If74b4b2a84811e266c8b12e70bf6bfe74ed4ea21
    Partial-Bug: #1487598

commit de604de334854e2eb6b4312ff57920564cbd4459
Author: OpenStack Proposal Bot <email address hidden>
Date: Sun Aug 30 01:39:06 2015 +0000

    Updated from global requirements

    Change-Id: Ie52aa3b59784722806726e4046bd07f4a4d97328

commit f0415ac20eaf5ab4abb9bd4839bf6d04ceee85d0
Author: armando-migliaccio <email address hidden>
Date: Fri Aug 28 13:53:04 2015 -0700

    Revert "Add support for unaddressed port"

    This implementation may expose a vulnerability where a malicious
    user can sieze the opportunity of a time window where a port
    may land unaddressed on a shared network, thus allowing him/her
    to suck up all the tenant traffic he/she wants....oh the shivers.

    This reverts commit d4c52b7f5a36a103a92bf9dcda7f371959112292.

    Change-Id: I7ebdaa8d3defa80eab90e460fde541a5bdd8864c

commit 013fdcd2a6d45dbe4de5d6e7077e5e9b60985ef9
Author: Assaf Muller <email address hidden>
Date: Fri Aug 28 16:41:07 2015 -0400

    Improve logging upon failure in iptables functional tests

    This will help us nail down a more accurate and efficient logstash
    query.

    Change-Id: Iee4238e358f7b056e373c7be8d6aa3202117a680
    Related-Bug: #1478847

commit 622dea818d851224a43d5276a81d5ce8a6eebb76
Author: Ivar Lazzaro <email address hidden>
Date: Mon Aug 17 17:17:42 2015 -0700

    handle gw_info outside of the db transaction on router creation

    Move the gateway interface creation outside the DB transaction
    to avoid lock timeout.

    Change-Id: I5a78d7f32e8ca912016978105221d5f34618af19
    Closes-bug: 1485809

commit 5b27d290a0a95f6247fc5a0fe6da1e7d905e6b2d
Author: Assaf Muller <email address hidden>
Date: Wed Aug 26 10:07:03 2015 -0400

    Remove ml2 resource extension success logging

    This is the cause of a tremendous amount of logs, for no
    perceivable gain. A normal dvr run in the gate shows this debug
    message around 120K times, which is way too much.

    Closes-Bug: #1489952

    Change-Id: I26fca8515d866a7cc1638d07fa33bc04479ae221

commit 8d3faf549cba2f58c872ef4121b2481e73464010
Author: huangpengtao <email address hidden>
Date: Fri Aug 28 23:20:46 2015 +0800

    Replace "prt" variable by "port"

    the local variable prt is meaningless,
    and port is used popular.

    Change-Id: I20849102cf5b4d84433c46791b4b1e2a22dc4739

commit ee374e7a5f4dea538fcd942f5...

tags: added: in-feature-pecan
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → liberty-3
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/179296
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-3 → 7.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/241710

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/241710
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=754a013611f1ca6bd5275c71562e81d2c9078945
Submitter: Jenkins
Branch: stable/kilo

commit 754a013611f1ca6bd5275c71562e81d2c9078945
Author: Assaf Muller <email address hidden>
Date: Mon Apr 13 13:26:06 2015 -0400

    Fix DVR interface delete by port when gateway is set

    When removing a DVR interface by port, the subnet_id
    passed to delete_csnat_router_interface_ports is None,
    and so it deletes all the DVR SNAT ports for the
    router.

    This patch fixes this issue by passing in the right
    subnet_id to the delete_csnat_router_interface_ports.

    Conflicts:
     neutron/db/l3_dvr_db.py
     neutron/tests/unit/db/test_l3_dvr_db.py

    Change-Id: I16735195c6575454876acd0e99ef45f382963566
    Closes-Bug: #1443524
    Co-Authored-By: Swaminathan Vasudevan <email address hidden>
    Co-Authored-By: Oleg Bondarev <email address hidden>
    (cherry picked from commit 0b5b6c7e746839d3f9bece6e782092bca3c9725e)

tags: added: in-stable-kilo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.