Comment 8 for bug 1787793

After the discussing during qos meeting last day [1], I think we have some work needs further study.
For the floating IPs bandwidth share, let me state some limitation of neutron first:
1. DVR scenario, floating IPs are hosting in different physical host, the bandwidth sharing now seems hard to achieve.
2. For one project, centralized floating IPs are located in the different router may still has no way to share the bandwidth. The router may schedule to different physical hosts too.
3. If you have mutiple external networks (floating IP networks), and each has its own NIC, then the share can not achieve. (p.s., neutron does not allow set mutiple gateway for router, so this seems not hurts so much.)
   Share floating IP bandwidth cross such external networks is not available.
4. So, the last one available scenario is floating IPs bandwidth hosted by one single centralized router.

So for the scenario 4, here are some reference TC classifier rules [2] [3] asked by Miguel Lavalle at the meeting.
Assuming we have 10Mbps sharing for IP: 1.1.1.1 and 2.2.2.2, so the rule will be:

tc qdisc add dev bond1 root handle 1: htb default 2
tc class add dev bond1 parent 1: classid 1:1 htb rate 10mbit ceil 10mbit # the share bandwidth, 10Mbps
tc class add dev bond1 parent 1: classid 1:2 htb rate 999mbit # the default class
tc class add dev bond1 parent 1:1 classid 1:10 htb rate 5mbit ceil 10mbit # IP 1 share rule
tc class add dev bond1 parent 1:1 classid 1:11 htb rate 5mbit ceil 10mbit # IP 2 share rule
tc filter add dev bond1 parent 1: protocol ip prio 1 u32 match ip src 1.1.1.1 flowid 1:10 # IP 1 filter match rule, egress
tc filter add dev bond1 parent 1: protocol ip prio 1 u32 match ip src 2.2.2.2 flowid 1:11 # IP 2 filter match rule, egress

For may test, I installed such rules to the pyhysic NIC bond1, and the neutron external network type is flat. Any other
network type, such as vlan, vxlan or gre, may change the tc filter protocol, such as 802.1q for vlan.

So, after installed these tc rules, the IP 1.1.1.1 and 2.2.2.2 can both work at 5Mbps. If 2.2.2.2 does not have data traffic,
then 1.1.1.1 can borrow the bandwidth from 2.2.2.2, it can reach the maximum 10Mbps.

[1] http://eavesdrop.openstack.org/meetings/neutron_qos/2018/neutron_qos.2018-08-28-15.01.log.html
[2] https://wiki.linuxfoundation.org/networking/ifb
[3] http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm