Inaccurate L3 QoS bandwidth

[LIU Yulong]

ENV:
stable/queens (12.0.1)
centos 7

Steps:
1. create a vm with floating IP 172.18.213.143
2. create qos policy with ingress and egress bandwidth limit rule:
    burst 10240 rate 10240
    aka, a 10mbps rule
3. run iperf3 in vm
4. iperf3 connect the vm via floating IP 172.18.213.143

iperf3 Testing:
the iperf output:
# iperf3 -c 172.18.213.143 -i 1
Connecting to host 172.18.213.143, port 5201
[ 4] local 172.18.211.30 port 36980 connected to 172.18.213.143 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 611 KBytes 5.00 Mbits/sec 54 2.83 KBytes
[ 4] 1.00-2.00 sec 752 KBytes 6.16 Mbits/sec 53 4.24 KBytes
[ 4] 2.00-3.00 sec 898 KBytes 7.35 Mbits/sec 57 4.24 KBytes
[ 4] 3.00-4.00 sec 604 KBytes 4.95 Mbits/sec 49 5.66 KBytes
[ 4] 4.00-5.00 sec 236 KBytes 1.93 Mbits/sec 23 9.90 KBytes
[ 4] 5.00-6.00 sec 810 KBytes 6.64 Mbits/sec 154 4.24 KBytes
[ 4] 6.00-7.00 sec 127 KBytes 1.04 Mbits/sec 27 5.66 KBytes
[ 4] 7.00-8.00 sec 509 KBytes 4.17 Mbits/sec 42 2.83 KBytes
[ 4] 8.00-9.00 sec 636 KBytes 5.21 Mbits/sec 48 4.24 KBytes
[ 4] 9.00-10.00 sec 127 KBytes 1.04 Mbits/sec 20 5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 5.19 MBytes 4.35 Mbits/sec 527 sender
[ 4] 0.00-10.00 sec 4.90 MBytes 4.11 Mbits/sec receiver

iperf Done.

The inaccurate bandwidth is because of the inappropriate tc filter mtu, default value is 2kb.
here is the filter:
=================================
filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 5675 success 5675)
  match IP src 172.18.213.143/32 (success 5675 )
 police 0xc rate 10240Kbit burst 1280Kb mtu 2Kb action drop overhead 0b
ref 1 bind 1

 Sent 404666 bytes 5675 pkts (dropped 0, overlimits 0)
=================================
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 33937 success 33937)
  match IP dst 172.18.213.143/32 (success 33937 )
 police 0xb rate 10240Kbit burst 1280Kb mtu 2Kb action drop overhead 0b
ref 1 bind 1

 Sent 51266073 bytes 34679 pkts (dropped 8268, overlimits 8268)
=================================

Solution:
set the mtu to 64kb
=================================
filter protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
  match IP dst 172.18.213.143/32 (success 0 )
 police 0xf rate 10240Kbit burst 1280Kb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
=================================
filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 (rule hit 0 success 0)
  match IP src 172.18.213.143/32 (success 0 )
 police 0x10 rate 10240Kbit burst 1280Kb mtu 64Kb action drop overhead 0b
ref 1 bind 1

 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
=================================

iperf testing:

Connecting to host 172.18.213.143, port 5201
[ 4] local 172.18.211.30 port 37886 connected to 172.18.213.143 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 2.49 MBytes 20.8 Mbits/sec 146 28.3 KBytes
[ 4] 1.00-2.00 sec 1.30 MBytes 10.9 Mbits/sec 119 7.07 KBytes
[ 4] 2.00-3.00 sec 1.24 MBytes 10.4 Mbits/sec 77 18.4 KBytes
[ 4] 3.00-4.00 sec 1.12 MBytes 9.38 Mbits/sec 72 19.8 KBytes
[ 4] 4.00-5.00 sec 1.12 MBytes 9.38 Mbits/sec 77 33.9 KBytes
[ 4] 5.00-6.00 sec 1.12 MBytes 9.38 Mbits/sec 100 39.6 KBytes
[ 4] 6.00-7.00 sec 1.18 MBytes 9.90 Mbits/sec 145 42.4 KBytes
[ 4] 7.00-8.00 sec 1.24 MBytes 10.4 Mbits/sec 141 32.5 KBytes
[ 4] 8.00-9.00 sec 1.43 MBytes 12.0 Mbits/sec 126 42.4 KBytes
[ 4] 9.00-10.00 sec 1.12 MBytes 9.38 Mbits/sec 80 24.0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 13.4 MBytes 11.2 Mbits/sec 1083 sender
[ 4] 0.00-10.00 sec 13.0 MBytes 10.9 Mbits/sec receiver

iperf Done.

[LIU Yulong (dragon889)]

Don't know why I remove the mtu parameters here:
https://review.openstack.org/#/c/453458/5..14/neutron/agent/linux/l3_tc_lib.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/576418

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576418
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5ddb2a4762eac52290cd0491881ec6ca15bb9217
Submitter: Zuul
Branch: master

commit 5ddb2a4762eac52290cd0491881ec6ca15bb9217
Author: LIU Yulong <email address hidden>
Date: Fri Jun 15 16:02:36 2018 -0400

    Make L3 IP tc filter rate limit more accurate

    Currently the L3 agent qos extension does not set the mtu for
    tc filter rules, the default value is 2kb. Tc filter rules use
    such mtu will result in an inaccurate bandwidth.

    So in order to improve the precision, we set the mtu to 64kb.
    For the test results, you can read the bug description.

    For more information you can read the linux tc man page:
    [1] https://linux.die.net/man/8/tc-tbf
    [2] http://man7.org/linux/man-pages/man8/tc-police.8.html

    Closes-Bug: #1777598
    Change-Id: I7a167ec6139ccc55988b34f82080a182116d02a3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/582158

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/582158
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c5e8d7743086f6c8906f1263cd941e4f47e344b8
Submitter: Zuul
Branch: stable/queens

commit c5e8d7743086f6c8906f1263cd941e4f47e344b8
Author: LIU Yulong <email address hidden>
Date: Fri Jun 15 16:02:36 2018 -0400

    Make L3 IP tc filter rate limit more accurate

    Currently the L3 agent qos extension does not set the mtu for
    tc filter rules, the default value is 2kb. Tc filter rules use
    such mtu will result in an inaccurate bandwidth.

    So in order to improve the precision, we set the mtu to 64kb.
    For the test results, you can read the bug description.

    For more information you can read the linux tc man page:
    [1] https://linux.die.net/man/8/tc-tbf
    [2] http://man7.org/linux/man-pages/man8/tc-police.8.html

    Closes-Bug: #1777598
    Change-Id: I7a167ec6139ccc55988b34f82080a182116d02a3
    (cherry picked from commit 5ddb2a4762eac52290cd0491881ec6ca15bb9217)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.4 release.