Neutron agent internal ports remain untagged for some time, which makes them trunk ports

Bug #1767422 reported by Miguel Angel Ajo on 2018-04-27
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Jakub Libosvar

Bug Description

Neutron agent ports are added to br-int without any tag. That makes them trunk ports (receiving traffic for all VLANs) until neutron-openvswitch-agent will handle them.

Sometimes the ports are left untagged forever, meaning that for example ha-router ha port will receive traffic directly from the external network (jumps to br-int to br-ex , and also back), or dnsmasq receives requests on the external network.

Outgoing traffic is dropped in br-ex though..

Vague details here (it's all we have so far):
This also becomes an issue (still under investigation) with the ovs-vswitchd agent and the revalidator thread (the thread that will check the kernel datapath flows under some circumstances to get stuck, for some reason it slows down a lot while analyzing trunk ports, eventually crashing the node on CPU usage).

This is also related to one security lp here: https://bugs.launchpad.net/bugs/1734320

Changed in neutron:
importance: Undecided → High
assignee: nobody → Miguel Angel Ajo (mangelajo)
milestone: none → rocky-1

Fix proposed to branch: master
Review: https://review.openstack.org/564825

Changed in neutron:
status: New → In Progress
description: updated
Changed in neutron:
assignee: Miguel Angel Ajo (mangelajo) → Slawek Kaplonski (slaweq)

Fix proposed to branch: master
Review: https://review.openstack.org/567225

Changed in neutron:
assignee: Slawek Kaplonski (slaweq) → Jakub Libosvar (libosvar)

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.openstack.org/567225
Reason: Using ovs-ofctl mod-port doesn't work on ports in namespaces.

Reviewed: https://review.openstack.org/564825
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88f5e11d8bf820b0124be0f6ec3c2d96011592d9
Submitter: Zuul
Branch: master

commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/566864
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd
Submitter: Zuul
Branch: stable/queens

commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)

tags: added: in-stable-queens

Reviewed: https://review.openstack.org/566865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=adb0ac4e5454391d68026cbeee93169578a10743
Submitter: Zuul
Branch: stable/pike

commit adb0ac4e5454391d68026cbeee93169578a10743
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Conflicts:
        neutron/tests/functional/agent/test_ovs_lib.py

        needed the addition of the following import:
    from neutron.plugins.ml2.drivers.openvswitch.agent.common import (
        constants as agent_const)

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)
    (cherry picked from commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd)

tags: added: in-stable-pike
tags: added: neutron-proactive-backport-potential

This issue was fixed in the openstack/neutron 13.0.0.0b2 development milestone.

This issue was fixed in the openstack/neutron 12.0.3 release.

This issue was fixed in the openstack/neutron 11.0.5 release.

Reviewed: https://review.openstack.org/567885
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e08233696816431b3f536bc556928491ecd14e2f
Submitter: Zuul
Branch: stable/ocata

commit e08233696816431b3f536bc556928491ecd14e2f
Author: Miguel Angel Ajo <email address hidden>
Date: Wed May 9 16:23:41 2018 +0200

    Don't delete flows on ports which were on dead vlan during plug

    Ocata codebase of the neutron agent deletes_flows
    when a port has been tagged and already had a tag.

    Later versions implement uninstall_flows to selectively delete
    specific flows, but such patches are big and buggy (have several
    follow up patches).

    This prevents that the patch handling 1767422 will get the DSCP
    flows deleted when port is tagged. Which is detected by functional
    testing.

    I have manually tested that setting a port admin_state_up False,
    and then True, will correctly move the port into dead vlan, and
    then back to non dead vlan, and properly remove the in_port=x,DROP
    openflow rule regardless of this change.

    Related: rhbz#1575706
    Related-Bug: 1767422

    Change-Id: Ib7915ae7bb7f471ff70ce25ce3beb16189ad5394

tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/567901
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=559bf87fd0d92e4d230058f5819c78f8b727d326
Submitter: Zuul
Branch: stable/ocata

commit 559bf87fd0d92e4d230058f5819c78f8b727d326
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Conflicts:
        neutron/tests/functional/agent/test_ovs_lib.py

        needed the addition of the following import:
    from neutron.plugins.ml2.drivers.openvswitch.agent.common import (
        constants as agent_const)

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)
    (cherry picked from commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.