Neutron agent internal ports remain untagged for some time, which makes them trunk ports

[Miguel Angel Ajo]

Neutron agent ports are added to br-int without any tag. That makes them trunk ports (receiving traffic for all VLANs) until neutron-openvswitch-agent will handle them.

Sometimes the ports are left untagged forever, meaning that for example ha-router ha port will receive traffic directly from the external network (jumps to br-int to br-ex , and also back), or dnsmasq receives requests on the external network.

Outgoing traffic is dropped in br-ex though..

Vague details here (it's all we have so far):
This also becomes an issue (still under investigation) with the ovs-vswitchd agent and the revalidator thread (the thread that will check the kernel datapath flows under some circumstances to get stuck, for some reason it slows down a lot while analyzing trunk ports, eventually crashing the node on CPU usage).

This is also related to one security lp here: https://bugs.launchpad.net/bugs/1734320

[Miguel Angel Ajo (mangelajo)]

Related to : https://bugzilla.redhat.com/show_bug.cgi?id=1558336

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/564825

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/566864

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/566865

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567225

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.openstack.org/567225
Reason: Using ovs-ofctl mod-port doesn't work on ports in namespaces.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/564825
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88f5e11d8bf820b0124be0f6ec3c2d96011592d9
Submitter: Zuul
Branch: master

commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/567885

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/567901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566864
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd
Submitter: Zuul
Branch: stable/queens

commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=adb0ac4e5454391d68026cbeee93169578a10743
Submitter: Zuul
Branch: stable/pike

commit adb0ac4e5454391d68026cbeee93169578a10743
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Conflicts:
        neutron/tests/functional/agent/test_ovs_lib.py

        needed the addition of the following import:
    from neutron.plugins.ml2.drivers.openvswitch.agent.common import (
        constants as agent_const)

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)
    (cherry picked from commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.5 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567885
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e08233696816431b3f536bc556928491ecd14e2f
Submitter: Zuul
Branch: stable/ocata

commit e08233696816431b3f536bc556928491ecd14e2f
Author: Miguel Angel Ajo <email address hidden>
Date: Wed May 9 16:23:41 2018 +0200

    Don't delete flows on ports which were on dead vlan during plug

    Ocata codebase of the neutron agent deletes_flows
    when a port has been tagged and already had a tag.

    Later versions implement uninstall_flows to selectively delete
    specific flows, but such patches are big and buggy (have several
    follow up patches).

    This prevents that the patch handling 1767422 will get the DSCP
    flows deleted when port is tagged. Which is detected by functional
    testing.

    I have manually tested that setting a port admin_state_up False,
    and then True, will correctly move the port into dead vlan, and
    then back to non dead vlan, and properly remove the in_port=x,DROP
    openflow rule regardless of this change.

    Related: rhbz#1575706
    Related-Bug: 1767422

    Change-Id: Ib7915ae7bb7f471ff70ce25ce3beb16189ad5394

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567901
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=559bf87fd0d92e4d230058f5819c78f8b727d326
Submitter: Zuul
Branch: stable/ocata

commit 559bf87fd0d92e4d230058f5819c78f8b727d326
Author: Miguel Angel Ajo <email address hidden>
Date: Fri Apr 27 18:05:48 2018 +0200

    Avoid agents adding ports as trunk by default.

    Agent OVS interface code adds ports without a vlan tag,
    if neutron-openvswitch-agent fails to set the tag, or takes
    too long, the port will be a trunk port, receiving
    traffic from the external network or any other port
    sending traffic on br-int.

    Also, those kinds of ports are triggering a code path
    on the ovs-vswitchd revalidator thread which can eventually
    hog the CPU of the host (that's a bug under investigation [1])

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1558336

    Conflicts:
        neutron/tests/functional/agent/test_ovs_lib.py

        needed the addition of the following import:
    from neutron.plugins.ml2.drivers.openvswitch.agent.common import (
        constants as agent_const)

    Co-Authored-By: Slawek Kaplonski <email address hidden>
    Change-Id: I024bbbdf7059835b2f23c264b48478c71633a43c
    Closes-Bug: 1767422
    (cherry picked from commit 88f5e11d8bf820b0124be0f6ec3c2d96011592d9)
    (cherry picked from commit 2b1d413ee90dfe2e9ae41c35ab37253df53fc6cd)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/848312

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/848312
Committed: https://opendev.org/openstack/neutron/commit/308924e5fb65359a6ccd6c0a6a26d7fda48aebee
Submitter: "Zuul (22348)"
Branch: master

commit 308924e5fb65359a6ccd6c0a6a26d7fda48aebee
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jul 6 07:42:33 2022 +0000

    Remove workaround for LP#1767422

    Since [1] and [2], the workaround done in [3] is no longer needed
    because we set the port tag when is bound and fixed the VLAN tag
    setting.

    [1]https://review.opendev.org/c/openstack/neutron/+/819567
    [2]https://review.opendev.org/c/openstack/neutron/+/820897
    [3]https://review.opendev.org/c/openstack/neutron/+/566864

    Closes-Bug: #1980126
    Related-Bug: #1767422

    Change-Id: Iee35cde7bbfee0e809cf71c4542dfbdefc97209f