[QoS][OvS] implement QoS bandwidth limitation by leveraging ovs meter

[LIU Yulong]

We are going to implement packet rate limit on ovs bridge by using meter rules [1], at the same time, meter can also be used to limit the bandwidth. os-key(ryu) supports the rule types of OFPMF_KBPS [2]. And usually, some smart-NICs for ovs offloading will support offloading meter rule to NIC to save the QoS cost of CPU on hosts naturally.

Since TC rules can be offloaded to smart-NIC as well. So this is an alternative to make the implementation of QoS limitation in a consistent mechanism.

thought?

[1] https://review.opendev.org/c/openstack/neutron/+/816800/8/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_int.py
[2] https://ryu.readthedocs.io/en/latest/ofproto_v1_3_ref.html#ryu.ofproto.ofproto_v1_3_parser.OFPMeterMod

[Akihiro Motoki (amotoki)]

This sounds a kind of RFE. Adding rfe tag for discussion.

[Lajos Katona (lajos-katona)]

We discussed this topic:
https://meetings.opendev.org/meetings/neutron_drivers/2022/neutron_drivers.2022-03-18-14.00.log.html#l-39

Based on it could you please elaborate the effects of this new QoS driver? Interesting questions are is metering rules perform better? Is the metering approach more accurate or such? How metering is better in offloaded case?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

First of all, sorry for not attending the drivers meeting, I was in PTO.

OF meters could be useful for max-BW ingress. Now we are using the interface traffic policies. We can change to OF meters instead.

For max-BW egress I would stay on QoS. I'm saying this because OF meters are not traffic shapers but rate limiters [1] (this link is old but accurate). This is like with the interface traffic rules: the traffic above the limit is dropped. If we move from QoS traffic shaper to OF meters, we'll reduce the quality of the current implementation.

For packet rate rules, because there is no way to implement them using OVS QoS, OF meters is the correct way.

One heads-up: OF meter action cannot be offloaded right now [2] (maybe you can provide updated information on this topic). For ML2 OVS, in Red Hat OSP16, we have implemented a workaround for max-BW egress and min-BW egress using "ip link" rate commands on the VF related to the OVS port representor connected.

Regards.

[1]https://opennetworking.org/wp-content/uploads/2013/04/openflow-spec-v1.3.1.pdf
[2]https://bugzilla.redhat.com/show_bug.cgi?id=2002406

[LIU Yulong (dragon889)]

@Lajos
Based on it could you please elaborate the effects of this new QoS driver?
It's very simple, a new bandwidth limitation with ovs meter will be added to current ovs qos_driver.

Interesting questions are is metering rules perform better?
Yes, it should be. Meter only take cares of rate limit, no traffic shaping.

Is the metering approach more accurate or such?
From our test results, it works fine, unified methods make things easy to use.

How metering is better in offloaded case?
Openvswitch can use one same API (rte_flow) to offload flows to smart NIC, including meter related flows and rules.

@Rodolfo,
Meter offloading support can be found here:
https://docs.nvidia.com/networking/display/MLNXOFEDv561033/Changes+and+New+Features

For my understanding, the ovs-agent bandwidth QoS driver is doing traffic rate limit only, no traffic shaping as well. So, meter rules for both packet rate and bandwith limit will be a better solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/860765

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/860766

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/860767

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/860761
Committed: https://opendev.org/openstack/neutron-lib/commit/99ba3d8b3f8fa94b7114cb8ba114d6d2fda7d683
Submitter: "Zuul (22348)"
Branch: master

commit 99ba3d8b3f8fa94b7114cb8ba114d6d2fda7d683
Author: LIU Yulong <email address hidden>
Date: Sat Oct 8 15:49:54 2022 +0800

    Add bandwidth rate limit table

    This new table is used to implement QoS bandwidth limitation
    by leveraging ovs meter.

    Closes-Bug: #1964342
    Change-Id: I87da4960a92bb807203534965bac2b0a42c5d4a4

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-lib 3.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860765
Committed: https://opendev.org/openstack/neutron/commit/c3ebefa5f73e8cb89c3c9c12b385b21e8749299f
Submitter: "Zuul (22348)"
Branch: master

commit c3ebefa5f73e8cb89c3c9c12b385b21e8749299f
Author: LIU Yulong <email address hidden>
Date: Fri Sep 2 10:13:20 2022 +0800

    Refactor for meter ID Generator

    Add a Singleton meter ID Generator for both bandwidth limit
    and packet rate limit, because for one bridge the meter ID
    is a sharing range.

    Closes-Bug: #1964342
    Change-Id: Ibb9762d57913ea701dcf2746a0e0db74c6a7ca01

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/866572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/866572
Committed: https://opendev.org/openstack/neutron/commit/fd423f7b1bd3e73631eb3588ea46140b0274ce7a
Submitter: "Zuul (22348)"
Branch: master

commit fd423f7b1bd3e73631eb3588ea46140b0274ce7a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Dec 1 16:26:25 2022 +0100

    Reset the MeterIDGenerator singleton after each test

    Closes-Bug: #1998474
    Related-Bug: #1964342
    Change-Id: I46853e65644d85ca0374b07a8d9e5816a82bcb36

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.