[RFE] QoS Explicit Congestion Notification (ECN) Support

[vikram.choudhary]

[Existing problem]
Network congestion can be very common in large data centers generating huge traffic from multiple hosts. Though each hosts can use IP header TOS ECN bit functionality to implement explicit congestion notification [1]_ but this will be a redundant effort.

[Proposal]
This proposal talks about achieving ECN on behalf of each host. This will help in making the solution centralized and can be done per tenant level. In addition to this traffic classification for applying ECN functionality can also be achieved via specific filtering rules, if required. Almost all the leading vendors support this option for better QoS [2]_.

Existing QoS framework is limited only to bandwidth rate limiting and be extend for supporting explicit congestion notification (RFC 3168 [3]_).

[Benefits]
- Enhancement to the existing QoS functionality.

[What is the enhancement?]
- Add ECN support to the QoS extension.
- Add additional command lines for realizing ECN functionality.
- Add OVS support.

[Related information]
[1] ECN Wiki
   http://en.wikipedia.org/wiki/Explicit_Congestion_Notification
[2] QoS
   https://review.openstack.org/#/c/88599/
[3] RFC 3168
   https://tools.ietf.org/html/rfc3168
[4] Specification
    https://blueprints.launchpad.net/neutron/+spec/explicit-congestion-notification
[5] Specification Discussion: https://etherpad.openstack.org/p/QoS_ECN
[6] OpenVSwitch support for ECN : http://openvswitch.org/support/dist-docs/ovs-ofctl.8.txt
[7] Etherpad Link : https://etherpad.openstack.org/p/QoS_ECN

[Miguel Angel Ajo (mangelajo)]

I'm not sure I get the full picture. Could you explain a little bit of a flow of how admin would configure this, what kinds of congestion would it detect, and how admin would see them, or what would happen when ECN bit is on?

[vikram.choudhary (vikschw)]

Will do an initial write up and upload a specification soon.

[Armando Migliaccio (armando-migliaccio)]

This has never happened, unless the tracking bot failed.

[Armando Migliaccio (armando-migliaccio)]

Please provide more details.

[Reedip (reedip-banerjee-deactivatedaccount)]

@Armando: Will provide a writeup for the same soon.

[vikram.choudhary (vikschw)]

Information captured @ https://beta.etherpad.org/p/Notepad

Thanks to Reedip for taking this up!

[Reedip (reedip-banerjee-deactivatedaccount)]

Updated link for review: https://etherpad.openstack.org/p/QoS_ECN

[Armando Migliaccio (armando-migliaccio)]

Ajo, Ihar: have you had the chance to have a first pass at the details provided?

[Miguel Angel Ajo (mangelajo)]

Sorry for the delay, I've added some questions to the etherpad.

It seems like it could be a reasonable enhancement to simple packet policing (note ingress policing still not supported, just egress).

My question on the etherpad:

#) Without a bandwidth limit rule, what's the source of congestion?, do we inspect the external interfaces to check the rate is too high?, do we inspect system load?
   System load could force ECN on low traffic instances
   System traffic rate could also force ECN on low traffic instances
   System traffic could be combined with the port packet counters to determine which ports are receiving more traffic, and act upon that.

I find this proposal interesting, but I think it's necessary to define:
1) how is "congestion" identified (there could be serveral forms, or to be tunable)
2) I think a reasonable way could be to create an ECN rule that identifies those parameters (a bandwidth_limit_rule_id -which has to be on the same policy-, or "host_load" / "host_traffic" .. etc - ? )
3) A high level description of how would it be implemented in the low level (I know, I contradict myself).

[Miguel Angel Ajo (mangelajo)]

@vikram, @reedip, can you answer to the questions on etherpad?

I propose we discuss this RFE on next QoS meeting to clarify.

[Miguel Angel Ajo (mangelajo)]

Ping, asked in qos-meeting, when can we talk about it?

[Reedip (reedip-banerjee-deactivatedaccount)]

Dear Ajo,
Just saw the logs from yesterday's meeting.
Sorry for the delay, we will put up our responses on the etherpad sheet itself

[Miguel Angel Ajo (mangelajo)]

One of the thoughts of the last meeting, was that, the high level seems clear now, but we may want to move this forward an have a spec to clarify the details of the implementation.

Specially we're not sure about how would you implement it in the low level.

@armax, if we could move this forward in the form of a spec, that'd be great.

[Akihiro Motoki (amotoki)]

I am not sure whether we really need to allow users to control ECN-enabled via API?
There are several levels of ECN support.
- virtual switches like OVS of compute nodes keeps ECN fields (i.e., do not reset ECN fields)
- set ECN fields based on a congestion level of virtual switches on compute nodes

I wonder what kind of things are exposed by ECN support.
Isn't it enough to support ECN in OVS on compute nodes based on the configuration?

(I added same questions to the etherpad too.)

I tried to follow the discussion on QoS meeting 3/23 but it seems the meeting bot did not log the conversation.
According to Miguel's commnent, QoS team now has clear high level view.
It would be great to share the high level view.

[Reedip (reedip-banerjee-deactivatedaccount)]

Akihiro:
The missing logs have been fished out and pasted here ( Lucky I have logging in my IRC :) )
http://paste.openstack.org/show/494363/

[Miguel Angel Ajo (mangelajo)]

We are moving this to incomplete until we have more feedback.

I had a talk with reedip during the summit, and it was clear that he wanted to leverage ECN, and how ECN works, but there was no clear picture of how the network node / the hypervisor would "detect" congestion, (by bw limit rules, by other means... etc).

[Reedip (reedip-banerjee-deactivatedaccount)]

Hi Miguel,
I have updated the https://etherpad.openstack.org/p/QoS_ECN Etherpad link for further discussion.
I would like your opinion on the same .

[Reedip (reedip-banerjee-deactivatedaccount)]

Updates: http://eavesdrop.openstack.org/meetings/neutron_qos/2017/neutron_qos.2017-01-31-15.02.log.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/445762

[Ihar Hrachyshka (ihar-hrachyshka)]

My belief is that qos subteam should make a judgement call here whether they are ready to support this initiative. It seems to me qos plate is already quite full.

[Kevin Benton (kevinbenton)]

Tentatively approved. Please write up a high-level spec because there are a lot of questions about implementation specifics.

[Kevin Benton (kevinbenton)]

I think it's just a matter of implementation details and reviewer resources

[Reedip (reedip-banerjee-deactivatedaccount)]

Similar proposal on https://bugs.launchpad.net/neutron/+bug/1708460

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/445762
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Reedip (<email address hidden>) on branch: master
Review: https://review.opendev.org/445762
Reason: Dont have sufficient time and bandwidth for this right now. If someone else wants to take it up, then it would be very helpful

[Lajos Katona (lajos-katona)]

As I see the history we can close this with won't fix if you think please reopen this RFE