[RFE] Minimum bandwidth support (egress)

[Miguel Angel Ajo]

Minimum bandwidth support (opposed to bandwidth limiting), guarantees a port minimum bandwidth when it's neighbours are consuming egress traffic and can be throttled in favor of the guaranteed port.

Strict minimum bandwidth support requires scheduling cooperation, to avoid physical interfaces overcommit. This RFE addresses only the hypervisor side of it. Scheduling cooperation will be addressed in a separate RFE [2] , this work is a pre-requisite for the 2nd step.

Use cases
=========

NFV/telcos are interested in this type of rules to make sure functions don't overcommit computes, and that any spawn of the same architecture will perform exactly as expected. This RFE is a prerequisite for [1].

Which in the mean time will provide a best effort guarantee on minimum bandwidth.

CSP could make use of it to provide guaranteed bandwidth for streaming, etc...

Notes
=====

Technologies like SR-IOV support that, and OVS & Linux bridge can be configured to support this type of service. Where in OvS it requires to use veth ports between bridges instead of patch ports, it introduces a performance overhead of a ~20%. Supporting this kind of rule for OvS agents must be made optional, so the administrators can choose it only when they really need it.

SR-IOV seems not to incur in any performance penalty.

This RFE title has been corrected to tackle only with instance-egress traffic, as per comments #1 and #2 of this rfe/bug, ingress is problematic, and even if it can be tackled, it's a much more complex beast, @armax knows about it [1]

[1] https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/supporting-network-bandwidth-guarantees-with-openstack-an-implementation-perspective

[2] https://bugs.launchpad.net/neutron/+bug/1578989

[Henry Gessau (gessau)]

I don't think it is possible to *guarantee* minimum bandwidth support for ingress. Let's say you have a 10Gb/s link and 10 VMs on a hypervisor. If one VM is receiving more than 1Gb/s of traffic, you can drop the excess but the dropped packets are still consuming link bandwidth and the other 9 VMs are left with <1Gb/s each.

[Miguel Angel Ajo (mangelajo)]

Hi henry, for egress it's "easy",
for ingress, it works "out of the box" with protocols that TCP which adapt to packet drop, since you are able to drop packets selectively, the specific TCP flows will addapt speed down.

UDP is a different beast, and requires inter-node communication or ECN signaling to tell the other end (or intermediate routers) to drop packets before they reach the node. It can be still done.

It's a good point, and in fact, I was talking about instance-egress here, so let's scope that for now, and let's look at instance-ingress in the future.

[Miguel Angel Ajo (mangelajo)]

For the strict/scheduling part, we may depend on the bigger nova plan: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086371.html

[Henry Gessau (gessau)]

Yes, for ingress you can do it with off-box support. However, then you need a driver to talk to the external entity, pluggable driver support (for all the vendors), and a reference implementation. It makes sense to have a separate RFE for ingress.

[Miguel Angel Ajo (mangelajo)]

Hi @rodolfo, thanks for stepping in as we discussed in the summit. I will be glad to help with reviews and code when we get this approved to go.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/316082

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

SPEC proposal: https://review.openstack.org/316082

I have started with the development of this feature. I'll submit the commit ASAP, although I'll put it as WIP.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/318531

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-05-26-22.01.log.html#l-107

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/344145

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347302

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316082
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=c5f9d0b0d403e3a3d0dd6557f58cc53f9aae5ba5
Submitter: Jenkins
Branch: master

commit c5f9d0b0d403e3a3d0dd6557f58cc53f9aae5ba5
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri May 13 13:22:56 2016 +0100

    Minimum bandwidth support (egress)

    Minimum bandwidth support (opposed to bandwidth limiting), guarantees
    a port minimum bandwidth when it's neighbours are consuming egress
    traffic and can be throttled in favor of the guaranteed port.

    Change-Id: Ia7a45f812f51d22469f5accf7efb3b515a656b64
    Related-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/351833

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/357865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/344145
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=60325f4ae9ec53734d792d111cbcf24270d57417
Submitter: Jenkins
Branch: master

commit 60325f4ae9ec53734d792d111cbcf24270d57417
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    Add QoS minimum bandwidth rule for instance egress traffic

    This patch introduces the front end implementation for QoS
    minimum bandwidth rule.

    APIImpact: New type of parameter for QoS rule in neutron API
    DocImpact

    Change-Id: I6b619a96a2bfde164646c71409b671352bc6ce7d
    Partial-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351833
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a2dc3c35e3d35c6a5f2099fee819e87b4fa216e9
Submitter: Jenkins
Branch: master

commit a2dc3c35e3d35c6a5f2099fee819e87b4fa216e9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    Add new configuration test in sanity check: vf_extended_management

    This test will check if 'ip link' version installed in this server
    supports extended VF management parameter 'min_tx_rate'. This
    parameter set the minimum egress rate for an interface.

    This test is executed when SR-IOV back-end and QoS extension
    are enabled.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: Ie9334f4ad2f6b047bf56689edf3333a8a612364a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347302
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=46de63c42e7c229529c0f101c9db8c84d73f6860
Submitter: Jenkins
Branch: master

commit 46de63c42e7c229529c0f101c9db8c84d73f6860
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    SR-IOV: add agent QoS driver to support egress minimum bandwidth

    This patch adds SR-IOV agent driver, which uses eswitch manager, to set
    VF min_tx_rate parameter. This parameter defines the guaranteed minimum
    bandwidth for egress traffic.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: Iefe5e698e99d186202d6ef170f84e93bfbba46dd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/364123

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/364123
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85bbeedbe0a83b42771258f117f2db9befa0a9d8
Submitter: Jenkins
Branch: master

commit 85bbeedbe0a83b42771258f117f2db9befa0a9d8
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Aug 31 18:48:14 2016 +0000

    qos: require min_kbps parameter for minimum bandwidth rule

    It does not have much physical sense to have a rule without the limit set, just
    direction. Require the parameter to avoid creating rules without the limit.

    Change-Id: I13c54be22f35ac7eb5835d8424a919d0b61a8e95
    Partial-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.openstack.org/318531

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/406841

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/357865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=84b3ae3ae9111e241352bdccad238c9fd37b83df
Submitter: Jenkins
Branch: master

commit 84b3ae3ae9111e241352bdccad238c9fd37b83df
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 16 08:50:35 2016 +0100

    Linux Bridge: driver support for QoS egress minimum bandwidth

    This patch provides the Linux Bridge agent driver the ability to control
    Linux Traffic Control (TC) to set the minimum required transmission rate
    for an interface.

    The TC library is refactored to use HTB qdiscs. This allows TC to
    define, for several flows in the same interface, the maximum and the
    minimum network bandwidth and the burst size.

    To be able to do traffic shaping (instead of policing) for ingress
    traffic, a new element, the Intermediate Functional Block device (IFB)
    is introduced.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: I4d4db54519f1435068d1af38819404d1e5d9cd52

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/415144
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/406841
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/406841
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Hirofumi Ichihara (<email address hidden>) on branch: master
Review: https://review.openstack.org/415144
Reason: Thank you for your review. I discussed in QoS meeting. We must make minimum bandwidth feature clear including Linuxbridge implementation. Now this looks like not on Pike cycle. I abandon once.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/620377

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620377
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=34910dc0745a26d8f5a7cbc01dc4ac84aa61c211
Submitter: Zuul
Branch: master

commit 34910dc0745a26d8f5a7cbc01dc4ac84aa61c211
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Nov 27 18:09:43 2018 +0000

    Add function helpers.compare_dict

    This function allows to compare two dicts, both keys and values.

    Change-Id: I0a68d627c00dde49b60e1b47d24822315c16d471
    Related-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/622859

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/622921

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/623268

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/624066

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/624437

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/624488

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.openstack.org/624488
Reason: Merged with https://review.openstack.org/#/c/624437/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624437
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=b934133e697527addc75849a99d811d90aec087b
Submitter: Zuul
Branch: master

commit b934133e697527addc75849a99d811d90aec087b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Dec 11 16:24:54 2018 +0000

    Add traffic control exceptions

    Added new traffic control exceptions TcLibQdiscNeededArguments
    and TcLibQdiscTypeError in ``neutron-lib.exceptions.qos``.

    Related-Bug: #1560963

    Change-Id: I4cad9bbfc5e0ed20bd24be33544ba0fa06898f54

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/625685

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/633951

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633951
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=f8b63272ff00d09cde40ee86f3357308cab53e00
Submitter: Zuul
Branch: master

commit f8b63272ff00d09cde40ee86f3357308cab53e00
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jan 30 14:02:43 2019 +0000

    Add traffic control constants

    Change-Id: If1902521086341f1f3786241509a000eaffe72b2
    Related-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/622859
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7cefa56e483e87ae4a46ebbf75dd759d919620c
Submitter: Zuul
Branch: master

commit d7cefa56e483e87ae4a46ebbf75dd759d919620c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Dec 11 19:15:35 2018 +0000

    Implement add_tc_qdisc and list_tc_qdiscs using pyroute2

    TcCommand.set_tbf_bw_limit() is used now to set and replace a TC TBF
    filter.

    Related-Bug: #1560963

    Change-Id: I162dea499d16db76692dd3d6d99b6be45f44ae59

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/622921
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=17a7a62e84b10ae8dfcc60df85862f90cb4d4725
Submitter: Zuul
Branch: master

commit 17a7a62e84b10ae8dfcc60df85862f90cb4d4725
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Dec 12 16:22:04 2018 +0000

    Implement delete_tc_qdisc using pyroute2

    Related-Bug: #1560963

    Change-Id: I79932dfc7464f9d904047df12d0f3f9cd5508a50

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/623268
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a92b0b1429cc2f823fbb32964d0519abfd1c6a1
Submitter: Zuul
Branch: master

commit 4a92b0b1429cc2f823fbb32964d0519abfd1c6a1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 10:14:51 2018 +0000

    Add add_tc_policy_class and list_tc_policy_classes using pyroute2

    Related-Bug: #1560963

    Change-Id: I01333077d4d47cf7f0d7ede4ccc5c2e2c63aa67a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/624066
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=006d4d07355eb5c77baf3fb1baa5c2cd22098766
Submitter: Zuul
Branch: master

commit 006d4d07355eb5c77baf3fb1baa5c2cd22098766
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 11:53:41 2018 +0000

    Add delete_tc_policy_class using pyroute2

    Related-Bug: #1560963

    Change-Id: Ide5b8975e00f98265da49fb3a4446e17c34af3bc

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/406841
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc8808d539e9b40fe8c7b9b9001f146edc663203
Submitter: Zuul
Branch: master

commit cc8808d539e9b40fe8c7b9b9001f146edc663203
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Nov 8 17:09:27 2018 +0000

    Add QoS minimum egress bandwidth rule into ovs-agent

    Add minimum egress bandwidth support for Open vSwitch.

    The scope of this implementation is reduced to N/S traffic.
    There is no QoS applied on traffic between VMs.

    The QoS rules are aplied to exit ports in bridges other than
    br-int; that means all physical bridges. No tunneled traffic
    will be shaped. This feature will be implemented in a following
    patch.

    Partial-Bug: #1560963
    Change-Id: I0a2ef52b13151a39e678e9a3e6f75babb47298d0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93d556e4344ee9ba25c51e2b72460c8ed69d9943
Submitter: Zuul
Branch: master

commit 93d556e4344ee9ba25c51e2b72460c8ed69d9943
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 12:32:06 2018 +0000

    Add TC filter functions implemented with pyroute2

    Added add_tc_filter_match_mac, add_tc_filter_policy and
    list_tc_filters

    Related-Bug: #1560963

    Change-Id: I360e68b98465706aef66e00590e1063345ead6b3

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/654403

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/655676

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/655969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/654403
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d099f17ebb95c53c69ed1777d8077f590bd5aeb
Submitter: Zuul
Branch: master

commit 5d099f17ebb95c53c69ed1777d8077f590bd5aeb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Apr 22 08:11:09 2019 +0000

    Add namespace support for "bridge" commands

    "bridge" commands executed inside a namespace will be needed
    initially to test the TC filter for VXLAN traffic. Those tests
    will create two namespaces with VXLAN interfaces in order to check
    the functionality of this new TC filter.

    Related-Bug: #1560963
    Change-Id: I3553b89fc0436c9cf83c66ab447ba4b4a6268ee1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=59e1796bd2ab04eef52bc1d5a4655edcd925bbf7
Submitter: Zuul
Branch: master

commit 59e1796bd2ab04eef52bc1d5a4655edcd925bbf7
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Apr 25 11:02:19 2019 +0000

    Retrieve statistics from TC policy classes

    Those statistics will be needed initially to test the TC filter
    for VXLAN traffic. Those tests will create several classes on the
    same interface with only one filter applied, diverting the traffic
    to only one of those classes. Once sample traffic matching the
    filter is injected, only the selected class should increase the
    packet/byte counters.

    Related-Bug: #1560963
    Change-Id: Ifc95051b67c031c0dfe209751df3d35c47d61148

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/655969
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5f99c79a504264397f3784abf295ba624241b380
Submitter: Zuul
Branch: master

commit 5f99c79a504264397f3784abf295ba624241b380
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 26 16:56:23 2019 +0000

    Add TC filtering for VXLAN traffic

    This new function in neutron.agent.linux.tc_lib creates a TC filter in
    a device depending on the VXLAN ID (VNI) and the source MAC address
    (usually the VM TAP MAC address). This filter will send all the egress
    tunneled traffic from the VM to a TC class in other to shape it (QoS).

    Change-Id: Ic04b52bc0aca7a18fa06ea89e981c80d67f42eb1
    Related-Bug: #1560963

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/668411

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668411
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7d62308eaa9fb4a80683499ea40f6d0c97fa7a4d
Submitter: Zuul
Branch: master

commit 7d62308eaa9fb4a80683499ea40f6d0c97fa7a4d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 1 10:27:06 2019 +0000

    Remove rootwrap filters for TC commands in Linux Bridge

    All neutron.agent.linux.tc_lib TC commands, used in Linux Bridge
    agent, have been implemented using Pyroute2.

    Change-Id: Idcac297b204900037b22ab25a516a161f4e78224
    Related-Bug: #1560963