[RFE] Minimum bandwidth support (egress)

Bug #1560963 reported by Miguel Angel Ajo on 2016-03-23
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Wishlist
Rodolfo Alonso

Bug Description

Minimum bandwidth support (opposed to bandwidth limiting), guarantees a port minimum bandwidth when it's neighbours are consuming egress traffic and can be throttled in favor of the guaranteed port.

Strict minimum bandwidth support requires scheduling cooperation, to avoid physical interfaces overcommit. This RFE addresses only the hypervisor side of it. Scheduling cooperation will be addressed in a separate RFE [2] , this work is a pre-requisite for the 2nd step.

Use cases
=========

NFV/telcos are interested in this type of rules to make sure functions don't overcommit computes, and that any spawn of the same architecture will perform exactly as expected. This RFE is a prerequisite for [1].

Which in the mean time will provide a best effort guarantee on minimum bandwidth.

CSP could make use of it to provide guaranteed bandwidth for streaming, etc...

Notes
=====

Technologies like SR-IOV support that, and OVS & Linux bridge can be configured to support this type of service. Where in OvS it requires to use veth ports between bridges instead of patch ports, it introduces a performance overhead of a ~20%. Supporting this kind of rule for OvS agents must be made optional, so the administrators can choose it only when they really need it.

SR-IOV seems not to incur in any performance penalty.

This RFE title has been corrected to tackle only with instance-egress traffic, as per comments #1 and #2 of this rfe/bug, ingress is problematic, and even if it can be tackled, it's a much more complex beast, @armax knows about it [1]

[1] https://www.openstack.org/summit/vancouver-2015/summit-videos/presentation/supporting-network-bandwidth-guarantees-with-openstack-an-implementation-perspective

[2] https://bugs.launchpad.net/neutron/+bug/1578989

Henry Gessau (gessau) on 2016-03-23
Changed in neutron:
status: New → Confirmed
importance: Undecided → Wishlist
tags: added: qos
Henry Gessau (gessau) wrote :

I don't think it is possible to *guarantee* minimum bandwidth support for ingress. Let's say you have a 10Gb/s link and 10 VMs on a hypervisor. If one VM is receiving more than 1Gb/s of traffic, you can drop the excess but the dropped packets are still consuming link bandwidth and the other 9 VMs are left with <1Gb/s each.

Miguel Angel Ajo (mangelajo) wrote :

Hi henry, for egress it's "easy",
for ingress, it works "out of the box" with protocols that TCP which adapt to packet drop, since you are able to drop packets selectively, the specific TCP flows will addapt speed down.

UDP is a different beast, and requires inter-node communication or ECN signaling to tell the other end (or intermediate routers) to drop packets before they reach the node. It can be still done.

It's a good point, and in fact, I was talking about instance-egress here, so let's scope that for now, and let's look at instance-ingress in the future.

summary: - [RFE] Minimum bandwidth support
+ [RFE] Minimum bandwidth support (ingress)
description: updated
description: updated
Miguel Angel Ajo (mangelajo) wrote :

For the strict/scheduling part, we may depend on the bigger nova plan: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086371.html

summary: - [RFE] Minimum bandwidth support (ingress)
+ [RFE] Minimum bandwidth support (egress)
Henry Gessau (gessau) wrote :

Yes, for ingress you can do it with off-box support. However, then you need a driver to talk to the external entity, pluggable driver support (for all the vendors), and a reference implementation. It makes sense to have a separate RFE for ingress.

tags: added: rfe
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
description: updated
Miguel Angel Ajo (mangelajo) wrote :

Hi @rodolfo, thanks for stepping in as we discussed in the summit. I will be glad to help with reviews and code when we get this approved to go.

description: updated

SPEC proposal: https://review.openstack.org/316082

I have started with the development of this feature. I'll submit the commit ASAP, although I'll put it as WIP.

Fix proposed to branch: master
Review: https://review.openstack.org/318531

Changed in neutron:
status: Confirmed → In Progress
Changed in neutron:
status: In Progress → Confirmed
Changed in neutron:
status: Confirmed → Triaged
tags: added: rfe-approved
removed: rfe
Changed in neutron:
milestone: none → newton-1
Changed in neutron:
status: Triaged → In Progress
Changed in neutron:
milestone: newton-1 → newton-2
Changed in neutron:
milestone: newton-2 → newton-3

Fix proposed to branch: master
Review: https://review.openstack.org/344145

Fix proposed to branch: master
Review: https://review.openstack.org/347302

Reviewed: https://review.openstack.org/316082
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=c5f9d0b0d403e3a3d0dd6557f58cc53f9aae5ba5
Submitter: Jenkins
Branch: master

commit c5f9d0b0d403e3a3d0dd6557f58cc53f9aae5ba5
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri May 13 13:22:56 2016 +0100

    Minimum bandwidth support (egress)

    Minimum bandwidth support (opposed to bandwidth limiting), guarantees
    a port minimum bandwidth when it's neighbours are consuming egress
    traffic and can be throttled in favor of the guaranteed port.

    Change-Id: Ia7a45f812f51d22469f5accf7efb3b515a656b64
    Related-Bug: #1560963

Fix proposed to branch: master
Review: https://review.openstack.org/357865

Changed in neutron:
assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) → David Shaughnessy (david-shaughnessy)
Changed in neutron:
assignee: David Shaughnessy (david-shaughnessy) → Ihar Hrachyshka (ihar-hrachyshka)
Changed in neutron:
assignee: Ihar Hrachyshka (ihar-hrachyshka) → Rodolfo Alonso (rodolfo-alonso-hernandez)

Reviewed: https://review.openstack.org/344145
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=60325f4ae9ec53734d792d111cbcf24270d57417
Submitter: Jenkins
Branch: master

commit 60325f4ae9ec53734d792d111cbcf24270d57417
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    Add QoS minimum bandwidth rule for instance egress traffic

    This patch introduces the front end implementation for QoS
    minimum bandwidth rule.

    APIImpact: New type of parameter for QoS rule in neutron API
    DocImpact

    Change-Id: I6b619a96a2bfde164646c71409b671352bc6ce7d
    Partial-Bug: #1560963

Reviewed: https://review.openstack.org/351833
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a2dc3c35e3d35c6a5f2099fee819e87b4fa216e9
Submitter: Jenkins
Branch: master

commit a2dc3c35e3d35c6a5f2099fee819e87b4fa216e9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    Add new configuration test in sanity check: vf_extended_management

    This test will check if 'ip link' version installed in this server
    supports extended VF management parameter 'min_tx_rate'. This
    parameter set the minimum egress rate for an interface.

    This test is executed when SR-IOV back-end and QoS extension
    are enabled.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: Ie9334f4ad2f6b047bf56689edf3333a8a612364a

Reviewed: https://review.openstack.org/347302
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=46de63c42e7c229529c0f101c9db8c84d73f6860
Submitter: Jenkins
Branch: master

commit 46de63c42e7c229529c0f101c9db8c84d73f6860
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 18 11:52:12 2016 +0100

    SR-IOV: add agent QoS driver to support egress minimum bandwidth

    This patch adds SR-IOV agent driver, which uses eswitch manager, to set
    VF min_tx_rate parameter. This parameter defines the guaranteed minimum
    bandwidth for egress traffic.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: Iefe5e698e99d186202d6ef170f84e93bfbba46dd

Fix proposed to branch: master
Review: https://review.openstack.org/364123

Changed in neutron:
assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) → Ihar Hrachyshka (ihar-hrachyshka)

Reviewed: https://review.openstack.org/364123
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85bbeedbe0a83b42771258f117f2db9befa0a9d8
Submitter: Jenkins
Branch: master

commit 85bbeedbe0a83b42771258f117f2db9befa0a9d8
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Aug 31 18:48:14 2016 +0000

    qos: require min_kbps parameter for minimum bandwidth rule

    It does not have much physical sense to have a rule without the limit set, just
    direction. Require the parameter to avoid creating rules without the limit.

    Change-Id: I13c54be22f35ac7eb5835d8424a919d0b61a8e95
    Partial-Bug: #1560963

Changed in neutron:
milestone: newton-3 → newton-rc1
Changed in neutron:
assignee: Ihar Hrachyshka (ihar-hrachyshka) → Hirofumi Ichihara (ichihara-hirofumi)
Changed in neutron:
assignee: Hirofumi Ichihara (ichihara-hirofumi) → Rodolfo Alonso (rodolfo-alonso-hernandez)
Changed in neutron:
status: In Progress → Fix Released

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.openstack.org/318531

Fix proposed to branch: master
Review: https://review.openstack.org/415144

Reviewed: https://review.openstack.org/357865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=84b3ae3ae9111e241352bdccad238c9fd37b83df
Submitter: Jenkins
Branch: master

commit 84b3ae3ae9111e241352bdccad238c9fd37b83df
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Aug 16 08:50:35 2016 +0100

    Linux Bridge: driver support for QoS egress minimum bandwidth

    This patch provides the Linux Bridge agent driver the ability to control
    Linux Traffic Control (TC) to set the minimum required transmission rate
    for an interface.

    The TC library is refactored to use HTB qdiscs. This allows TC to
    define, for several flows in the same interface, the maximum and the
    minimum network bandwidth and the burst size.

    To be able to do traffic shaping (instead of policing) for ingress
    traffic, a new element, the Intermediate Functional Block device (IFB)
    is introduced.

    DocImpact
    Partial-Bug: #1560963

    Change-Id: I4d4db54519f1435068d1af38819404d1e5d9cd52

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/415144
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/406841
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

tags: added: neutron-proactive-backport-potential
Changed in neutron:
milestone: newton-rc1 → pike-1
status: Fix Released → In Progress

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/406841
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
milestone: pike-1 → pike-2

Change abandoned by Hirofumi Ichihara (<email address hidden>) on branch: master
Review: https://review.openstack.org/415144
Reason: Thank you for your review. I discussed in QoS meeting. We must make minimum bandwidth feature clear including Linuxbridge implementation. Now this looks like not on Pike cycle. I abandon once.

Changed in neutron:
assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) → nobody
Changed in neutron:
milestone: pike-2 → none
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)

Reviewed: https://review.openstack.org/620377
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=34910dc0745a26d8f5a7cbc01dc4ac84aa61c211
Submitter: Zuul
Branch: master

commit 34910dc0745a26d8f5a7cbc01dc4ac84aa61c211
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Nov 27 18:09:43 2018 +0000

    Add function helpers.compare_dict

    This function allows to compare two dicts, both keys and values.

    Change-Id: I0a68d627c00dde49b60e1b47d24822315c16d471
    Related-Bug: #1560963

Related fix proposed to branch: master
Review: https://review.openstack.org/622921

Related fix proposed to branch: master
Review: https://review.openstack.org/623268

Related fix proposed to branch: master
Review: https://review.openstack.org/624066

Related fix proposed to branch: master
Review: https://review.openstack.org/624488

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.openstack.org/624488
Reason: Merged with https://review.openstack.org/#/c/624437/

Reviewed: https://review.openstack.org/624437
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=b934133e697527addc75849a99d811d90aec087b
Submitter: Zuul
Branch: master

commit b934133e697527addc75849a99d811d90aec087b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Dec 11 16:24:54 2018 +0000

    Add traffic control exceptions

    Added new traffic control exceptions TcLibQdiscNeededArguments
    and TcLibQdiscTypeError in ``neutron-lib.exceptions.qos``.

    Related-Bug: #1560963

    Change-Id: I4cad9bbfc5e0ed20bd24be33544ba0fa06898f54

Reviewed: https://review.openstack.org/633951
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=f8b63272ff00d09cde40ee86f3357308cab53e00
Submitter: Zuul
Branch: master

commit f8b63272ff00d09cde40ee86f3357308cab53e00
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jan 30 14:02:43 2019 +0000

    Add traffic control constants

    Change-Id: If1902521086341f1f3786241509a000eaffe72b2
    Related-Bug: #1560963

Reviewed: https://review.openstack.org/622859
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7cefa56e483e87ae4a46ebbf75dd759d919620c
Submitter: Zuul
Branch: master

commit d7cefa56e483e87ae4a46ebbf75dd759d919620c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Dec 11 19:15:35 2018 +0000

    Implement add_tc_qdisc and list_tc_qdiscs using pyroute2

    TcCommand.set_tbf_bw_limit() is used now to set and replace a TC TBF
    filter.

    Related-Bug: #1560963

    Change-Id: I162dea499d16db76692dd3d6d99b6be45f44ae59

Reviewed: https://review.openstack.org/622921
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=17a7a62e84b10ae8dfcc60df85862f90cb4d4725
Submitter: Zuul
Branch: master

commit 17a7a62e84b10ae8dfcc60df85862f90cb4d4725
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Dec 12 16:22:04 2018 +0000

    Implement delete_tc_qdisc using pyroute2

    Related-Bug: #1560963

    Change-Id: I79932dfc7464f9d904047df12d0f3f9cd5508a50

Reviewed: https://review.openstack.org/623268
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4a92b0b1429cc2f823fbb32964d0519abfd1c6a1
Submitter: Zuul
Branch: master

commit 4a92b0b1429cc2f823fbb32964d0519abfd1c6a1
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 10:14:51 2018 +0000

    Add add_tc_policy_class and list_tc_policy_classes using pyroute2

    Related-Bug: #1560963

    Change-Id: I01333077d4d47cf7f0d7ede4ccc5c2e2c63aa67a

Reviewed: https://review.openstack.org/624066
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=006d4d07355eb5c77baf3fb1baa5c2cd22098766
Submitter: Zuul
Branch: master

commit 006d4d07355eb5c77baf3fb1baa5c2cd22098766
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 11:53:41 2018 +0000

    Add delete_tc_policy_class using pyroute2

    Related-Bug: #1560963

    Change-Id: Ide5b8975e00f98265da49fb3a4446e17c34af3bc

Reviewed: https://review.openstack.org/406841
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc8808d539e9b40fe8c7b9b9001f146edc663203
Submitter: Zuul
Branch: master

commit cc8808d539e9b40fe8c7b9b9001f146edc663203
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Nov 8 17:09:27 2018 +0000

    Add QoS minimum egress bandwidth rule into ovs-agent

    Add minimum egress bandwidth support for Open vSwitch.

    The scope of this implementation is reduced to N/S traffic.
    There is no QoS applied on traffic between VMs.

    The QoS rules are aplied to exit ports in bridges other than
    br-int; that means all physical bridges. No tunneled traffic
    will be shaped. This feature will be implemented in a following
    patch.

    Partial-Bug: #1560963
    Change-Id: I0a2ef52b13151a39e678e9a3e6f75babb47298d0

Reviewed: https://review.openstack.org/625685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93d556e4344ee9ba25c51e2b72460c8ed69d9943
Submitter: Zuul
Branch: master

commit 93d556e4344ee9ba25c51e2b72460c8ed69d9943
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 13 12:32:06 2018 +0000

    Add TC filter functions implemented with pyroute2

    Added add_tc_filter_match_mac, add_tc_filter_policy and
    list_tc_filters

    Related-Bug: #1560963

    Change-Id: I360e68b98465706aef66e00590e1063345ead6b3

Related fix proposed to branch: master
Review: https://review.opendev.org/655676

Related fix proposed to branch: master
Review: https://review.opendev.org/655969

Reviewed: https://review.opendev.org/654403
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d099f17ebb95c53c69ed1777d8077f590bd5aeb
Submitter: Zuul
Branch: master

commit 5d099f17ebb95c53c69ed1777d8077f590bd5aeb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Apr 22 08:11:09 2019 +0000

    Add namespace support for "bridge" commands

    "bridge" commands executed inside a namespace will be needed
    initially to test the TC filter for VXLAN traffic. Those tests
    will create two namespaces with VXLAN interfaces in order to check
    the functionality of this new TC filter.

    Related-Bug: #1560963
    Change-Id: I3553b89fc0436c9cf83c66ab447ba4b4a6268ee1

Reviewed: https://review.opendev.org/655676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=59e1796bd2ab04eef52bc1d5a4655edcd925bbf7
Submitter: Zuul
Branch: master

commit 59e1796bd2ab04eef52bc1d5a4655edcd925bbf7
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Apr 25 11:02:19 2019 +0000

    Retrieve statistics from TC policy classes

    Those statistics will be needed initially to test the TC filter
    for VXLAN traffic. Those tests will create several classes on the
    same interface with only one filter applied, diverting the traffic
    to only one of those classes. Once sample traffic matching the
    filter is injected, only the selected class should increase the
    packet/byte counters.

    Related-Bug: #1560963
    Change-Id: Ifc95051b67c031c0dfe209751df3d35c47d61148

Reviewed: https://review.opendev.org/655969
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5f99c79a504264397f3784abf295ba624241b380
Submitter: Zuul
Branch: master

commit 5f99c79a504264397f3784abf295ba624241b380
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Apr 26 16:56:23 2019 +0000

    Add TC filtering for VXLAN traffic

    This new function in neutron.agent.linux.tc_lib creates a TC filter in
    a device depending on the VXLAN ID (VNI) and the source MAC address
    (usually the VM TAP MAC address). This filter will send all the egress
    tunneled traffic from the VM to a TC class in other to shape it (QoS).

    Change-Id: Ic04b52bc0aca7a18fa06ea89e981c80d67f42eb1
    Related-Bug: #1560963

Reviewed: https://review.opendev.org/668411
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7d62308eaa9fb4a80683499ea40f6d0c97fa7a4d
Submitter: Zuul
Branch: master

commit 7d62308eaa9fb4a80683499ea40f6d0c97fa7a4d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 1 10:27:06 2019 +0000

    Remove rootwrap filters for TC commands in Linux Bridge

    All neutron.agent.linux.tc_lib TC commands, used in Linux Bridge
    agent, have been implemented using Pyroute2.

    Change-Id: Idcac297b204900037b22ab25a516a161f4e78224
    Related-Bug: #1560963

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related blueprints