Deleting a VM port does not remove Security rules in ip tables

Bug #1333365 reported by chandrasekaran natarajan
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Elena Ezhova
Kilo
New
Undecided
Unassigned

Bug Description

Deleting a VM port does not remove security rules associated to VM port in ip tables.

Setup :

ICEHOUSE GA with KVM Compute node,network node, controller

1. Spawn a VM with security group attached.
2. Delete a VM port
3. Verify the ip tables

VM IP : 10.10.1.4
Rules attached : TCP and icmp rule

root@ICN-KVM:~# ovs-vsctl show
f3b34ea5-9799-460d-99bb-26359fd26e38
    Bridge "br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "phy-br-eth1"
            Interface "phy-br-eth1"
        Port "eth1"
            Interface "eth1"
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "qvof28b18dc-c3" <<<<<<<<<<<<<<<<<<< VM tap port
            tag: 1
            Interface "qvof28b18dc-c3"
        Port "int-br-eth1"
            Interface "int-br-eth1"
    ovs_version: "2.0.1"
root@ICN-KVM:~#

After Deleting a port security rules are still present in iptables.
---------------------------------------------------------------------

oot@ICN-KVM:~# iptables-save | grep 28b18dc
:neutron-openvswi-if28b18dc-c - [0:0]
:neutron-openvswi-of28b18dc-c - [0:0]
:neutron-openvswi-sf28b18dc-c - [0:0]
-A neutron-openvswi-FORWARD -m physdev --physdev-out tapf28b18dc-c3 --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-FORWARD -m physdev --physdev-in tapf28b18dc-c3 --physdev-is-bridged -j neutron-openvswi-sg-chain
-A neutron-openvswi-INPUT -m physdev --physdev-in tapf28b18dc-c3 --physdev-is-bridged -j neutron-openvswi-of28b18dc-c
-A neutron-openvswi-if28b18dc-c -m state --state INVALID -j DROP
-A neutron-openvswi-if28b18dc-c -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-if28b18dc-c -p tcp -m tcp -j RETURN
-A neutron-openvswi-if28b18dc-c -p icmp -j RETURN
-A neutron-openvswi-if28b18dc-c -s 10.10.1.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A neutron-openvswi-if28b18dc-c -j neutron-openvswi-sg-fallback
-A neutron-openvswi-of28b18dc-c -p udp -m udp --sport 68 --dport 67 -j RETURN
-A neutron-openvswi-of28b18dc-c -j neutron-openvswi-sf28b18dc-c
-A neutron-openvswi-of28b18dc-c -p udp -m udp --sport 67 --dport 68 -j DROP
-A neutron-openvswi-of28b18dc-c -m state --state INVALID -j DROP
-A neutron-openvswi-of28b18dc-c -m state --state RELATED,ESTABLISHED -j RETURN
-A neutron-openvswi-of28b18dc-c -j RETURN
-A neutron-openvswi-of28b18dc-c -j neutron-openvswi-sg-fallback
-A neutron-openvswi-sf28b18dc-c -s 10.10.1.4/32 -m mac --mac-source FA:16:3E:D4:47:F8 -j RETURN
-A neutron-openvswi-sf28b18dc-c -j DROP
-A neutron-openvswi-sg-chain -m physdev --physdev-out tapf28b18dc-c3 --physdev-is-bridged -j neutron-openvswi-if28b18dc-c
-A neutron-openvswi-sg-chain -m physdev --physdev-in tapf28b18dc-c3 --physdev-is-bridged -j neutron-openvswi-of28b18dc-c
root@ICN-KVM:~#

Changed in neutron:
assignee: nobody → akash (akashg1611)
Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/110805

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/110805
Reason: This change is old enough and hasn't seen any updates since July 30, 2014. Abandoning it, please revive it if you plan to work on it again.

Revision history for this message
Elena Ezhova (eezhova) wrote :

This bug is still valid and can considered as a security vulnerability. Since the work on the fix has stopped 5 months ago, I am assigning this bug to myself.

@Akash, if you are planning to continue your work on this bug, please state it in comments and feel free to reassign the bug to yourself.

Changed in neutron:
assignee: Akash Gangil (akashg1611) → nobody
assignee: nobody → Elena Ezhova (eezhova)
Kyle Mestery (mestery)
Changed in neutron:
milestone: none → kilo-2
importance: Undecided → Medium
Kyle Mestery (mestery)
Changed in neutron:
milestone: kilo-2 → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/143865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d6a55c17360d1aa8ca91849199987ae71e8600ee
Submitter: Jenkins
Branch: master

commit d6a55c17360d1aa8ca91849199987ae71e8600ee
Author: Elena Ezhova <email address hidden>
Date: Wed Dec 24 17:09:32 2014 +0300

    Remove port from ovsdb after its deletion

    When port is being deleted via API it is not removed
    from ovsdb and corresponding iptables chains remain
    even though the port does not exist.

    This patch adds a notification for the ovs neutron agent,
    upon which the port is deleted from ovsdb.

    Co-Authored-By: Akash Gangil <email address hidden>
    Closes-Bug: #1333365
    Change-Id: Iccda3bee98d561ef3a06d0317d3d68d6b1dfb76b

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-3 → 2015.1.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/179314

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/179314
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e007167a700aa5b80ecb48adff0ac36bb330591d
Submitter: Jenkins
Branch: master

commit e007167a700aa5b80ecb48adff0ac36bb330591d
Author: Kevin Benton <email address hidden>
Date: Thu Apr 30 17:14:44 2015 -0700

    Don't delete port from bridge on delete_port event

    Commit d6a55c17360d1aa8ca91849199987ae71e8600ee added
    logic to the OVS agent to delete a port from the integration
    bridge when a port was deleted on the Neutron side. However,
    this led to several races where whoever created the initial
    port (e.g. Nova, L3 agent, DHCP agent) would be trying to
    remove the port from the bridge at the same time. These
    would result in ugly exceptions on one side or the other.

    The original commit was trying to address the problem where
    the port would maintain connectivity even though it was removed
    from the integration bridge.

    This patch addresses both cases by removing the iptables rules
    for the deleted port and putting it in the dead VLAN so it loses
    connectivity. However, it still leaves the port attached to the
    integration bridge so the original creator can delete it.

    Related-Bug: #1333365
    Closes-Bug: #1448148
    Change-Id: I7ae7750b7ac7d15325ed9f2d517ca171543b53be

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/187795

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/178666
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0ace88fd4a75ff213dc36fd16c1f8e7080ab7d6d
Submitter: Jenkins
Branch: master

commit 0ace88fd4a75ff213dc36fd16c1f8e7080ab7d6d
Author: Robert Li <email address hidden>
Date: Fri May 8 11:08:45 2015 -0400

    Add VIF_DELETED notification event to Nova

    It's possible to delete a neutron port that is currently associated
    with an instance. When it happens, neutron should notify nova of the
    port deletion event so that Nova can take proper actions.

    Refer to I998b6bb80cc0a81d665b61b8c4a424d7219c666f for the nova patch
    that handles the event.

    Change-Id: Iff88cd12ae18017ef3e776821bcf3ecf3b4f052f
    Related-Bug: #1333365
    Related-Bug: #1448148

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/187795
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=81e043f72135682510727c9fa9bafe7efa676717
Submitter: Jenkins
Branch: stable/kilo

commit 81e043f72135682510727c9fa9bafe7efa676717
Author: Kevin Benton <email address hidden>
Date: Thu Apr 30 17:14:44 2015 -0700

    Don't delete port from bridge on delete_port event

    Commit d6a55c17360d1aa8ca91849199987ae71e8600ee added
    logic to the OVS agent to delete a port from the integration
    bridge when a port was deleted on the Neutron side. However,
    this led to several races where whoever created the initial
    port (e.g. Nova, L3 agent, DHCP agent) would be trying to
    remove the port from the bridge at the same time. These
    would result in ugly exceptions on one side or the other.

    The original commit was trying to address the problem where
    the port would maintain connectivity even though it was removed
    from the integration bridge.

    This patch addresses both cases by removing the iptables rules
    for the deleted port and putting it in the dead VLAN so it loses
    connectivity. However, it still leaves the port attached to the
    integration bridge so the original creator can delete it.

    Conflicts:
     neutron/plugins/openvswitch/agent/ovs_neutron_agent.py
     neutron/tests/unit/plugins/openvswitch/agent/test_ovs_neutron_agent.py
     neutron/tests/unit/plugins/openvswitch/test_ovs_tunnel.py

    Related-Bug: #1333365
    Closes-Bug: #1448148
    Change-Id: I7ae7750b7ac7d15325ed9f2d517ca171543b53be
    (cherry picked from commit e007167a700aa5b80ecb48adff0ac36bb330591d)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (feature/pecan)

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (feature/pecan)

Change abandoned by Kyle Mestery (<email address hidden>) on branch: feature/pecan
Review: https://review.openstack.org/196701
Reason: This is lacking the functional fix [1], so I'll propose a new merge commit which includes that one.

[1] https://review.openstack.org/#/c/196711/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (feature/pecan)

Related fix proposed to branch: feature/pecan
Review: https://review.openstack.org/196920

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (feature/pecan)
Download full text (171.5 KiB)

Reviewed: https://review.openstack.org/196920
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7f759c077f8f860c13db92d2ea6b353ef6b70900
Submitter: Jenkins
Branch: feature/pecan

commit 8123144fadd7c5d5e6e56a76ea860512619a2cf6
Author: Moshe Levi <email address hidden>
Date: Sun Jun 28 14:37:14 2015 +0300

    Fix Consolidate sriov agent and driver code

    This patch add mising __init to mech_sriov/mech_driver/
    and update the setup.cfg to the new agent entrypoint

    Trivial Fix

    Change-Id: I53a527081feb78472f496675bbb3c5121d38a14a

commit 8942fccf02e6e179d47582fdb2792a1ca972da21
Author: Assaf Muller <email address hidden>
Date: Mon Jun 29 11:38:51 2015 -0400

    Remove failing SafeFixture tests

    The fixtures 1.3 release attempted to fix the fixtures resource
    leak issue, but failed to do so completely. Our own SafeFixture
    is still needed: The 1.3 release broke our SafeFixture tests,
    but not the usage of SafeFixture itself. This patch removes
    those failing tests for now to unbreak the gate. Jakub reported
    a bug on fixtures 1.3:
    https://bugs.launchpad.net/python-fixtures/+bug/1469759

    We will continue to use SafeFixture until that bug is fixed
    in fixtures, at which point we will be able to require
    fixtures > 1.3.

    Change-Id: I59457c3bb198ff86d5ad55a1e623d008f0034b8f
    Closes-Bug: #1469734

commit 71dffb0a2c1720cd8233a329d32958a0160dd6f5
Author: Kevin Benton <email address hidden>
Date: Mon Jun 29 08:27:41 2015 +0000

    Revert "Removed test_lib module"

    This reverts commit 9a6536de6e1a7fe9b2552adc142e254426b82b6f.

    We pulled all of the plugins out of the tree, many of which still inherit
    from neutron test classes. This change then stated that we no longer
    support testing other plugins. I think this is a bit premature and should
    have been discussed under the subject
    "Neutron plugins can't use neutron plugin unit tests" or something
    similar.

    Change-Id: I68318589f010b731574ea3bfa8df98492bab31fc

commit b20fd81dbd497e058384a0af065dd0f1fdc4c728
Author: Jakub Libosvar <email address hidden>
Date: Fri Jun 5 14:32:51 2015 +0000

    Refactor NetcatTester class

    Following capabilities were added:
       - used transport protocol is passed as a constant instead of bool
       - src port for testing was added
       - connection can be established explicitly
       - change constructor parameters of NetcatTester

    As a part of removing bool for protocol definition
    get_free_namespace_port() was also modified to match the behavior.

    Change-Id: Id2ec322e7f731c05a3754a65411c9a5d8b258126

commit 83e37980dcd0b2bad6d64dd2cb23bcd2891cafca
Author: jingliuqing <email address hidden>
Date: Sat Jun 27 13:41:54 2015 +0800

    Use REST rather than ReST

    Change-Id: I06c9deaab58c5ec13bfeec39fb8fd4b1fe21f42d

commit 1b60df85ba3ad442c2e4e7e52538e1b9a1bf9378
Author: Kevin Benton <email address hidden>
Date: Thu Jun 25 18:34:38 2015 -0700

    Add a double-mock guard to the base test case

    Use mock to patch mock with a check to prevent multiple active
    patches to the...

tags: added: in-feature-pecan
Revision history for this message
Rex Wang (wzhengxian) wrote :

I have met the similar problems although my setup is with HAVANA on Ubuntu12.04:

1. I find following logs " No such device " from /var/log/openvswitch/ovs-vswithces.log:
2015-12-14T06:13:49.647Z|00085|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-58538441-9d device failed: No such device
2015-12-14T06:13:49.657Z|00086|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qg-58538441-9d device failed: No such device
2015-12-14T06:13:49.793Z|00087|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-c64f0f6b-08 device failed: No such device
2015-12-14T06:13:49.800Z|00088|netdev_linux|WARN|ioctl(SIOCGIFINDEX) on qr-c64f0f6b-08 device failed: No such device
2015-12-14T06:13:49.879Z|00089|bridge|INFO|bridge br-ex: added interface qg-d6a925fc-73 on port 1710
2015-12-14T06:13:49.879Z|00090|netdev_linux|WARN|qg-58538441-9d: removing policing failed: No such device
2015-12-14T06:13:49.879Z|00091|netdev_linux|WARN|qr-c64f0f6b-08: removing policing failed: No such device
2015-12-14T06:13:50.211Z|00092|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-d6a925fc-73 device failed: No such device
2015-12-14T06:13:50.667Z|00093|bridge|INFO|bridge br-ex: added interface qg-8cec2368-ab on port 1711
2015-12-14T06:13:50.779Z|00094|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:13:50.986Z|00095|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-8cec2368-ab device failed: No such device
2015-12-14T06:13:51.161Z|00096|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:07.893Z|00097|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:08.368Z|00098|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:20.632Z|00099|connmgr|INFO|br-eth3<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:20.727Z|00100|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:20.878Z|00101|bridge|INFO|bridge br-int: added interface tap8fe04af5-5d on port 3625
2015-12-14T06:34:21.152Z|00102|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2015-12-14T06:34:21.197Z|00103|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tap8fe04af5-5d device failed: No such device
2015-12-14T06:34:21.206Z|00104|netdev_linux|WARN|Dropped 4 log messages in last 1231 seconds (most recently, 12

2. ports in OVS not deleted even after all things deleted from neutron/nova parts.
I create some number of VMs and then delete them. Do such thing repeatedly for a period of time, then I find some ports in OVS on network node not deleted(All VMs deleted).

And can I implemente following commit on my network node:
Reviewed: https://review.openstack.org/143865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d6a55c17360d1aa8ca91849199987ae71e8600ee
Submitter: Jenkins
Branch: master

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.