[RFE] Create L3 floating IPs with qos (rate limit)

[LiuYong]

For a certain tenant, there is not a implementation used for limiting the egrees bandwidth of floating ips.

Use cases
=========
In certain commercial scenery with neutron, such as some VNF cases using TECS of ZTE, the max egrees bandwidth of a floating ip should be limited because of its expensive cost and its flexibility with different fixed ips at different envirenments. As for the bandwidth limiting for one floating ip, it should be the implementation by means of the parameter '--max-kbps' in '--qos-policy-rule'. But for now, it is not allowed to executing 'floating-ip-create' with '--qos-policy'.

Now like below:
============
[root@localhost devstack]# neutron floatingip-create public --qos-policy qos-policy
Unrecognized attribute(s) 'qos_policy'
Neutron server returns request_ids: ['req-b7ff870c-3f43-424b-a694-703726090d69']

I think it should be:
=====================
When a floating ip, created with a 'qos-policy', is associated with a fixed ip, the 'qos-policy' is applied on the fixed port accordingly. This implementation will bring more convenience when tenant destroy the fixed port and deploy another re-associating.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/337969

[OpenStack Infra (hudson-openstack)]

Change abandoned by LiuYong (<email address hidden>) on branch: master
Review: https://review.openstack.org/337969
Reason: Another is valid

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/338000

[Armando Migliaccio (armando-migliaccio)]

This seems a genuine request but I'd defer to Ihar or Ajo and the rest of the folks interested in QoS to see if they can take the time to crunch this through the review machine.

That said I wonder why the qos policy needs to be applied to the FIP, can't that still be enforced to the FIXED IP associated to it? That won't require any API changes. Or am I barking up at the wrong tree?

[LiuYong (liu-yong8-deactivatedaccount)]

This propose, I think, mainly want to provide the ablity for the admin, not to control the within openstack traffic but south-north traffic of certain VM, because of real cost of the latter.

[Armando Migliaccio (armando-migliaccio)]

I am having a hard time parsing your comment.

A floating IP is associated to a port, that port can have QoS rules on. The admin can see and manage that port, qos included. Besides the implementation, what else is missing? Wouldn't the API abstraction suffice as it is?

[Ihar Hrachyshka (ihar-hrachyshka)]

It was discussed on one of qos meetings: http://eavesdrop.openstack.org/meetings/neutron_qos/2016/neutron_qos.2016-06-29-14.00.log.html

Not sure how to link there, so:

"14:35:19 <njohnston> A QoS question came up in the mailing list. I haven't had the bandwidth to really think about it, but I wanted to see if anyone wanted to respond.
14:35:24 <njohnston> #link http://lists.openstack.org/pipermail/openstack-dev/2016-June/098207.html
14:36:20 <ihrachys> hm, qg maintains all FIPs at the same time right?
14:36:20 <ajo_> yes, to implement that we need to do it on the routers
14:36:37 <ihrachys> so you can't just TC on the gq port
14:37:03 <ajo_> or, in the DVR implementation details...
14:37:14 <ajo_> but yes, doing it on the vm ports won't help
14:37:48 <ihrachys> well we could apply FIP port policies on relevant VM ports, but that is twisted behaviour for my taste
14:37:55 <ihrachys> because it will still limit internal traffic
14:38:40 <ihrachys> a router would probably need to apply some advanced traffic classification to slow down traffic destined for specific fixed ips
14:40:50 <njohnston> OK, good, I thought that the answer was not totally straightforward.
14:41:32 <njohnston> Should we encourage the author to file this as a bug so we can discuss it in more detail in launchpad?
14:42:49 * njohnston fears we lost ajo_ for good.
14:43:03 <ihrachys> njohnston: I think we should suggest RFE
14:43:04 <ajo> njohnston, may be an RFE makes sense here
14:43:10 <ajo> to keep us all thinking about it"

Basically, if we want to limit traffic destined to a fixed ip on router, we would need to do advanced traffic classification. I believe TC work that is currently in progress but will potentially take several cycles is a dependency to even consider the feature. I don't know if we have a mechanism in RFE process to postpone. If we do, that's what I would choose.

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-07-21-22.00.log.html#l-297

[Armando Migliaccio (armando-migliaccio)]

Deferred until we can lay down the foundation.

@Ihar: do you still expect API changes required once these foundations are laid?

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/338000
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

[LIU Yulong (dragon889)]

I have a draft RFE spec, here: https://github.com/gotostack/gotostack.github.io/blob/master/pages/CloudComputing/layer-3-rate-limit.rst.
As you can see in that spec, I'd prefer adding a new `rate_limit` attribute to those L3 IPs.
Then we will expand the floating IP resource (extension), and router gateway, at the same time the DB models. Which benefit us is that it's really simple and efficient, users could directly know the bandwidth of L3 IPs. Without other side effects, and we can easily set a default value for this l3 IPs bandwidth.

And in the QoS meeting, those nice guys suggest to use the policy attachment:
http://eavesdrop.openstack.org/meetings/neutron_qos/2016/neutron_qos.2016-09-21-14.03.log.html
This will be another solution.

Any feedback, advice or suggestions are welcomed. Thanks.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/374506

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/424466

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/424468

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/374506
Reason: Comments on bug report seems to hint this effort has been abandoned. Please resume, if you intend to proceed.

[LIU Yulong (dragon889)]

Since the spec and the patches were submitted there, here we change the RFE title to make those more match. And hope the neutron cores or neutron qos guys may pay some attention to this RFE. I'll raise this to the next neutron team meeting or neutron_qos meeting. And I've added this to the neutron-ptg-pike: https://etherpad.openstack.org/p/neutron-ptg-pike. sincerely hope this can be accepted to pike. Thank you guys.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/453458

[Ihar Hrachyshka (ihar-hrachyshka)]

I believe the proposal is moving in the right direction (I am on spec review). There is already a set of WIP patches that show feasibility of implementation using tc. Seems like a useful feature.

Of course, the final buy-in should come from qos subteam.

[OpenStack Infra (hudson-openstack)]

Change abandoned by LIU Yulong (<email address hidden>) on branch: master
Review: https://review.openstack.org/424468
Reason: Please see the change: https://review.openstack.org/#/c/425218/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/374506
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=9b9d1e9a8823cdb7b5a8687f1eaefd4b2e1d701b
Submitter: Jenkins
Branch: master

commit 9b9d1e9a8823cdb7b5a8687f1eaefd4b2e1d701b
Author: LIU Yulong <email address hidden>
Date: Sun Sep 18 13:56:19 2016 +0800

    Layer 3 IPs rate limit

    The patch proposes a general solution for the L3 IP QoS,which adds
    `qos_policy_id` to the L3 IPs. Then the QoS rules will be used to
    set the linux tc rules in router namespace efficiently. For the L3
    users, the new attribute would indicate the bandwidth they can use.

    Change-Id: If7dba69857a6758ef278905cd10434b1de4447b5
    Related-Bug: #1596611

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/453458
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f40128b437e58e39efc9f02af7135c02f7826b42
Submitter: Zuul
Branch: master

commit f40128b437e58e39efc9f02af7135c02f7826b42
Author: LIU Yulong <email address hidden>
Date: Wed Apr 5 09:38:55 2017 +0800

    [L3][QoS] Adding L3 rate limit TC lib

    This is the TC lib utils for L3 IP QoS implementation.
    For more detail please see [1]: L3 agent side TC rules.

    [1] https://review.openstack.org/#/c/374506/

    Partially-Implements blueprint: floating-ip-rate-limit
    Related-Bug: #1596611
    Change-Id: Icfec83ca6dc31d7283d9c6c6ef0997d5e60daae6

[OpenStack Infra (hudson-openstack)]

Change abandoned by Paul Belanger (<email address hidden>) on branch: master
Review: https://review.openstack.org/424466
Reason: We are dealing with an issue in zuulv3 today, as such we need to abandon / restore this patch to correct the issue. Apologies for the inconvenience.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424466
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8fcda21a0634aa24fedb9c3e482ecfd495883c9b
Submitter: Zuul
Branch: master

commit 8fcda21a0634aa24fedb9c3e482ecfd495883c9b
Author: LIU Yulong <email address hidden>
Date: Tue Apr 26 17:24:11 2016 +0800

    [L3][QoS] Neutron server side Floating IP QoS

    Add qos_policy_id to floating IP, then the user could set/update
    the binding QoS policy of a floating IP.

    APIImpact: a new parameter qos_policy_id was added to floating IP
    related API.

    Partial-Bug: #1596611
    Partially-Implements blueprint: floating-ip-rate-limit
    Change-Id: I4efe9e49d268dffeb3df4de4ea1780152218633b

[LIU Yulong (dragon889)]

All floating IPs QoS related patches can be found here:
https://review.openstack.org/#/q/topic:bp/floating-ip-rate-limit

Also see blueprint:
https://blueprints.launchpad.net/neutron/+spec/floating-ip-rate-limit

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424468
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=00bf365025772d423eea7da1ddbb6211ab757aed
Submitter: Zuul
Branch: master

commit 00bf365025772d423eea7da1ddbb6211ab757aed
Author: LIU Yulong <email address hidden>
Date: Thu Apr 28 13:17:08 2016 +0800

    [L3][QoS] Neutron server side router gateway IP QoS

    This patch enables to bind a QoS policy to the router gateway,
    then in L3 agent side SNAT traffic for the VMs without floating
    IPs can be limited under the policy bandwidth rules. This is
    suit for all kinds of L3 routers: DVR, DVR with SNAT HA, L3 HA
    and Legacy.

    API update router gateway json:
    {
    router": {
      "external_gateway_info": {
        ...
        "qos_policy_id": "policy-uuid"
        }
      }
    }

    Depends-On: https://review.openstack.org/#/c/567497/

    Partially-Implements blueprint: router-gateway-ip-qos
    Closes-Bug: #1757044
    Related-Bug: #1596611
    Change-Id: I26e22bce7edd1f93b2ac0048b61b14f858938537

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/658451

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722415
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e7e71b2ca67169e6de4cdad71f2c82059132325d
Submitter: Zuul
Branch: master

commit e7e71b2ca67169e6de4cdad71f2c82059132325d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Apr 21 14:53:34 2020 +0000

    [OVN] Implement floating IP QoS in OVN backend

    This patch implements in the OVN backend the existing floating
    IP QoS extension.

    The OVN client, using the existing QoS extension, will retrieve
    the QoS rules attached to each floating IP, the router where the
    floating IP lives and the router gateway port. The QoS rules
    will be applied on the router gateway port.

    The OVN NB QoS rules for floating IP addresses have a "match"
    field containing a tuple of parameters:
    - The direction of the flow:
        'inport == "src"' or
        'outport == "dst"'
    - The IP address to match:
        'ip4.src == 1.2.3.4' or
        'ip4.dst == 1.2.3.4'
    - The chassis where the port is located:
        'is_chassis_resident("chassis")'

    Closes-Bug: #1877408
    Related-Bug: #1596611
    Depends-On: https://review.opendev.org/#/c/727847/

    Change-Id: Ib65d8edcb0a415f6d698c952334d3b4bb0d9fff6

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/817797

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/817797
Committed: https://opendev.org/openstack/neutron-lib/commit/8736420eeafd3ed9fa0199381ba80e235c08eccd
Submitter: "Zuul (22348)"
Branch: master

commit 8736420eeafd3ed9fa0199381ba80e235c08eccd
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Fri Nov 12 17:33:31 2021 +0000

    Rehome QoS floating IP extension

    Related-Bug: #1596611
    Change-Id: Ida281a0a4fa8001e117f4803c55314595bc06c18